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Abstract 

The Pierre Auger Observatory, located on a vast, high plain in western Argentina, is the 
world’s largest cosmic ray observatory. The objectives of the Observatory are to probe 
the origin and characteristics of cosmic rays above 10 17 eV and to study the interactions 
of these, the most energetic particles observed in nature. The Auger design features an 
array of 1660 water Cherenkov particle detector stations spread over 3000 km 2 over¬ 
looked by 24 air fluorescence telescopes. In addition, three high elevation fluorescence 
telescopes overlook a 23.5 km 2 , 61-detector infilled array with 750 m spacing. The Ob¬ 
servatory has been in successful operation since completion in 2008 and has recorded 
data from an exposure exceeding 40,000 km 2 sr yr. This paper describes the design and 
performance of the detectors, related subsystems and infrastructure that make up the 
Observatory. 

Keywords: Pierre Auger Observatory, high energy cosmic rays, hybrid observatory, 
water Cherenkov detectors, air fluorescence detectors 
PACS: 95.55.Vj, 95.85.Ry, 96.50.sd, 98.70.Sa 
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1. Introduction 

The origin of high energy cosmic rays has been a mystery since the discovery of 
extensive air showers in the late 1930s [1-5]. In 1962, John Linsley recorded a cosmic 
ray event with an energy of 10 20 eV [6]. Subsequent work found more events near and 
above 10 20 eV [7-10]. The Pierre Auger Observatory was proposed to discover and 
understand the source or sources of cosmic rays with the highest energies. 

A unique partnership of 18 countries, the Pierre Auger Collaboration came together 
to pursue this science. Construction of the Pierre Auger Observatory was started in 
2002 and completed in 2008. The purpose of this article is to review the design and 
performance of the detector systems and associated infrastructure that constitute the 
Observatory. 

To achieve the scientific goals, the Collaboration designed the Pierre Auger Ob¬ 
servatory for a high statistics study of cosmic rays at the highest energies. Measured 
properties of the air showers determine the energy and arrival direction of each cos¬ 
mic ray. These properties also provide a statistical determination of the distribution 
of primary masses (cosmic ray composition). The Pierre Auger Observatory in the 
Province of Mendoza, Argentina, has been taking data since 2004, adding detectors as 
they became active until completion in 2008. 

The Observatory is a hybrid detector, a combination of a large surface detector (SD) 
and a fluorescence detector (FD). The SD is composed of a baseline array, comprising 
1660 water Cherenkov stations placed in a triangular grid with nearest neighbors sepa¬ 
rated by 1500 m, and a smaller array (stations separated by 750 m). The surface array 
is spread over an area of ~ 3000 km 2 , and is depicted in Figure 1. Figure 2 shows 
examples of FD (left) and SD (right) detector elements. This area is generally flat, 
with detectors located at altitudes between 1340 m and 1610 m. The mean altitude is 
~ 1400 m, corresponding to an atmospheric overburden of ~ 875 g cm -2 . The array is 
located between latitudes 35.0° and 35.3° S and between longitudes 69.0° and 69.4° W. 

As an aid to the reader we provide here a list of acronyms used throughout this 
paper 1 . 


1 AERA, Auger Engineering Radio Array; AMIGA, Auger Muon and Infilled Ground Array; APF, 
Aerosol phase function; ARQ, Automatic repeat request; BLS, Balloon Launching Station; CDAS, Central 
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Figure 1: The Auger Observatory. Each dot corresponds to one of the 1660 surface detector stations. The 
four fluorescence detector enclosures are shown, each with the 30° field of view of its six telescopes. Also 
shown are the two laser facilities, CLF and XLF, near the Observatory center. 



Figure 2: The fluorescence detector enclosure Los Leones (top); a surface detector station (bottom). 


Data Acquisition System; CIC, Constant Intensity Cut method; CLF, Central Laser Facility; EAS, Extensive 
air shower; ELVES: (defined in Section 14.2); FADC, Flash Analog-to-Digital Convertor; FD, Fluorescence 
Detector; FRAM, Photometric Robotic Atmospheric Monitor; GDAS, Global Data Assimilation System; 
HAM, Horizontal Attenuation Monitor; HEAT, High-Elevation Auger Telescopes; LDF, Lateral distribu¬ 
tion function; MD, Muon Detector; PLD, Programmable logic device; PMT, Photomultiplier tube; SCS, 
Slow Control System; SD, Surface Detector; SDP, Shower-detector plane; TDMA, Time Division Multiple 
Access; ToT, Time over threshold; VEM, Vertical Equivalent Muon; XLF, extreme Laser Facility 
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1.1. Highlights of science results 

The Pierre Auger Observatory has made key measurements of the highest energy 
cosmic rays. Cosmic ray showers with zenith angle < 60° are defined as vertical show¬ 
ers, while those with 60° < zenith angle < 80° are defined as horizontal showers. The 
energy spectrum is measured with unprecedented precision using four datasets: hybrid 
(events measured simultaneously by the SD and FD), SD 750 m array, SD baseline 
vertical, and SD baseline horizontal [11, 12], Thanks to the high statistics of the SD 
data, a first harmonic analysis was performed in different energy ranges starting from 
2.5 x 10 17 eV in a search for dipolar modulations in right ascension [13, 14]. The up¬ 
per limits in the dipole amplitude impose stringent constraints in astrophysical models 
[13, 15, 16], 

The Auger data provide evidence for a weak correlation between arrival directions 
of cosmic rays above 55 EeV and the positions of AGNs with z < 0.018 in the VCV 
catalogue [17, 18]. The Collaboration also has performed the measurement of the 
proton-air cross-section at 57 TeV [19] that favors a moderately slow rise of the cross- 
section towards higher energies, and inferred the proton-proton cross-section, whose 
value is within one sigma of the best extrapolation from the recent LHC data points 
[20], The composition measurements could be interpreted as an evolution from light 
to heavier nuclei if current hadronic interaction models describe well the air shower 
physics [21-24], 

Upper limits have been obtained on the photon flux integrated above an energy 
threshold which impose stringent limits for top-down models [25, 26]. Also, compet¬ 
itive neutrino limits were published [27-29], as well as searches for Galactic neutron 
signals [30, 31]. 

1.2. Observatory design 

Design targets for the surface detector array included 100% duty cycle, a well- 
defined aperture independent of energy above 10 18 5 eV, measurement of the time struc¬ 
ture of the signals of the shower particles, sensitivity to showers arriving at large zenith 
angles, self-contained detector stations and in situ calibration of detector stations by 
cosmic ray muons. The fluorescence detector design required that every event above 
10 19 eV arriving within the FD on-time should be recorded by at least one fluores¬ 
cence telescope camera, direct measurement of the longitudinal development profile 
and timing synchronization for simultaneous measurement of showers with the surface 
detector array [32], 

Each water Cherenkov surface detector is self-powered and communicates with 
the central data acquisition system using wireless technology. Air fluorescence tele¬ 
scopes record air shower development in the atmosphere above the surface array on 
dark moonless nights. There are four ah' fluorescence sites on the perimeter of the 
array, each with six telescopes. 

An essential feature of this Auger hybrid design is the capability of observing air 
showers simultaneously by two different but complementary techniques. The SD op¬ 
erates continuously, measuring the particle densities as the shower strikes the ground 
just beyond its maximum development. On dark nights, the FD records the develop¬ 
ment of the air shower via the amount of nitrogen fluorescence produced along its path. 
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Since the intensity of fluorescence light is proportional to the energy dissipated by the 
shower, integrating the intensity of light produced along the shower axis yields a nearly 
calorimetric measurement of the cosmic ray energy. Using the observation of hybrid 
events, this energy calibration can then be transferred to the surface array with its 100% 
duty factor and large event gathering power. Moreover, independent measurements by 
the surface array and the fluorescence detectors alone have limitations that can be over¬ 
come by comparing their measurements in the set of showers measured by both. The 
hybrid dataset provides a thorough understanding of the capabilities and the systematic 
uncertainties of both components. 

The FD always operates in conjunction with the SD. Its primary purpose is to mea¬ 
sure the longitudinal profile of showers recorded by the SD whenever it is dark and clear 
enough to make reliable measurements of atmospheric fluorescence from air showers. 
The integral of the longitudinal profile is used to determine the shower energy, and the 
rate of shower development is indicative of the primary particle’s mass. The hybrid 
detector has better angular resolution than the surface array by itself. 

Subsequent to the completion of construction, two significant enhancements have 
been incorporated into the baseline detectors that significantly extend the Observatory’s 
science capability. The HEAT (High Elevation Auger Telescopes) fluorescence detec¬ 
tors together with a 750 m array, part of AMIGA (Auger Muon and Infilled Ground 
Array) extend the sensitivity down to 10 17 eV, in keeping with the hybrid detection 
strategy of the original Observatory. 

1.3. Background 

The Pierre Auger Observatory was conceived during the International Cosmic Ray 
Conference in Dublin in 1991 by Jim Cronin of the University of Chicago and Alan 
Watson of the University of Leeds. It had become clear to them that only the construc¬ 
tion of a very large air shower array would yield the statistical power and complete sky 
coverage necessary to address the question of the origin of the highest energy cosmic 
rays. 

A six-month design workshop was held in 1995 that produced a Design Report [32] 
with a discussion of the science, a conceptual design and cost estimate. The design re¬ 
port became the basis for funding proposals by the collaborating countries. Subsequent 
to the workshop a team of scientists evaluated numerous prospective sites in both hemi¬ 
spheres. Preferred sites were selected in the southern and northern hemispheres by the 
collaboration in 1995 and 1996, respectively. At the direction of the funding agencies, 
the project was to begin by building the Observatory in the southern hemisphere. 

After a period of research and development, the Engineering Array, consisting of 
32 prototype surface array detectors and two prototype fluorescence telescopes, was 
built to validate the design [33], At the end of 2001, before the end of the scheduled 
two years, the Engineering Array was able to record and reconstruct air shower events 
simultaneously by the surface array and the fluorescence detectors. The Engineering 
Array demonstrated the validity of the design and the performance of all of the detec¬ 
tor systems, communications equipment and data acquisition as well as the deployment 
methodology. The detectors performed better than the design requirements, substan¬ 
tially increasing the physics reach of the Observatory. 
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Installation of production detectors began in 2002. While the Engineering Array 
was assembled and deployed almost completely by Auger collaborators, production 
deployment was accomplished by trained Observatory staff. Scientists monitored the 
quality of the work and carried out the commissioning of completed detectors. The 
Observatory started collecting data in January 2004 with 154 active detector stations. 
The first physics results were presented during the 2005 summer conference season. 

Many important results have now been published by the Auger Collaboration that 
have had a major impact on the held of cosmic ray physics. As of this writing, 60 
full author list papers have been published or accepted, with another 2 submitted and 
about 7 more in preparation. The Auger Collaboration is also training a cadre of future 
scientists, with 238 students granted PhDs based on their work on Auger. Another 161 
PhD students are in the pipeline. Publications and other technical reports are available 
online at http : //www. auger . org/technical_inf o/. 

2. Hybrid design 

As indicated above, a key feature of the Pierre Auger Observatory is its hybrid 
design, in which ultrahigh energy cosmic rays are detected simultaneously by a sur¬ 
face array and by fluorescence telescopes. The two techniques are used to observe air 
showers in complementary ways, providing important cross-checks and measurement 
redundancy [34, 35]. 

The surface detector array views a slice of an air shower at ground level, with robust 
and sensitive water Cherenkov stations which respond to both the electromagnetic and 
muonic components of the shower. Well-established methods exist for determining 
arrival directions and for estimating primary energy (see Section 1 1). The SD operates 
24 hours per day and thus provides uniform coverage in right ascension with a huge 
3000 km 2 collecting area. The instantaneous aperture of the array is easily calculable, 
especially for energies above 3 x 10 18 eV, where a shower falling on any part of the 
array is detected with 100% efficiency independently of the mass of the primary particle 
that initiated the shower. The aperture is found simply by counting the number of 
hexagons of active surface stations at any time, and multiplying by the aperture, AQ., 
of a hexagonal cell, 4.59km 2 sr (for shower zenith angles <60°) [36]. The SD has 
the important property that the quality of the measurements improves with the shower 
energy. 

The fluorescence detector is used to image the longitudinal development of the 
shower cascade in the atmosphere. The fluorescence light is emitted isotropically in 
the ultraviolet part of the spectrum and is produced predominantly by the electromag¬ 
netic component of the shower. Observation periods are limited to dark nights of good 
weather, representing a duty cycle which has increased from 12% during early years 
[37] up to ~15% at the present time (see Section 12). This disadvantage is balanced 
by the considerable gain of being able to view the development of the shower pro¬ 
file. Firstly, since fluorescence light production is proportional to the collisional en¬ 
ergy deposit in the atmosphere, the technique provides a near-calorimetric method for 
determining the primary cosmic ray energy. Secondly, the depth at which a shower 
reaches maximum size, 2f max , is observable: this is the most direct of all accessible 
mass composition indicators. 
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The aim is to use the FD and the SD to measure the same properties of primary 
cosmic rays (energy, mass composition, direction) but to do so using different tech¬ 
niques with very different systematic uncertainties. Thus, part of the function of the 
fluorescence detector is to enable cross-checks to be made and to train the surface ar¬ 
ray, providing confidence in the SD measurements made during the majority of the 
time when no fluorescence detector is operating. However, the fluorescence detector is 
much more than a calibration tool. The data set collected during hybrid observations 
is of high quality, being especially useful for those studies that require more precise 
shower directions than are available from the surface array alone and for studies where 
longitudinal profile measurements are vital. 

An example of the synergy between the two techniques is illustrated by measure¬ 
ments of the cosmic ray energy spectrum for showers arriving with zenith angles less 
than 60° [38], The exposure achievable with the surface array is much larger than is 
possible with hybrid measurements so that in principle a greater number of events can 
be used in the determination of the spectrum. However, with the surface array alone, 
there is a serious problem in that the relationship between the primary energy and the 
SD observable chosen to mirror it, namely the signal measured in the water Cherenkov 
detectors at 1000 m from the shower axis, 5(1000), can only be found using cascade 
simulations. This method is inherently unreliable as the necessary hadronic physics 
is unknown at the energies of interest and it is therefore not even practical to assign 
a reliable systematic uncertainty. Using the hybrid system, it has proved possible to 
develop an alternative method for estimating the primary energy that is essentially free 
from simulations. 

The first step is to quantify the dependence of 5(1000) with zenith angle. This 
is done using the “constant intensity” method [39], where the attenuation of the typi¬ 
cal air shower with increasing atmospheric depth is mapped out using SD data alone. 
The conversion to primary energy is then achieved using a subset of SD events also 
observed with the FD. The only simulation input to the determination of primary en¬ 
ergy with the FD is in estimating the small fraction (~10%) that goes into neutrinos 
and high energy muons that continue into the ground. Atmospheric variability (mostly 
changing aerosol properties) complicates the analysis, due to essential corrections for 
atmospheric attenuation of the fluorescence light and because of the allowance that 
must be made for scattered Cherenkov light in the FD signal. The event reconstruc¬ 
tions utilize extensive atmospheric monitoring that is performed at the Observatory site 
whenever the FD operates. The final step in measuring the energy spectrum is a precise 
determination of the exposure (km 2 sryr) for the observations. As already mentioned, 
the instantaneous aperture of the SD array is straightforward to calculate, even during 
the period of construction when it was continually growing. This example of an anal¬ 
ysis procedure illustrates the complementary strengths of the SD and FD, and how a 
robust result can be achieved by drawing on them. 

A key to the success of the hybrid technique is that it allows a precise determination 
of the position of a shower axis in space with an accuracy better than could be achieved 
independently with either the surface array detectors or a single fluorescence telescope 
[35], The first step in geometrical reconstruction makes use of the known orientations 
of the pixels of the fluorescence detector and of the light intensities registered at the 
pixels. This enables the shower-detector plane (SDP), the plane in space that contains 
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the shower axis and the FD site, to be determined. Timing information is then used 
to find the orientation of the shower axis within the SDP (see Section 10). With a 
FD alone, the accuracy of determining the shower geometry may be poor if the angular 
length of the observed shower track is short, say less than 15°. In this case, the apparent 
angular speed of the shower image in the telescope is approximately constant, leading 
to a degeneracy in the geometry solution. This degeneracy is broken if the angular 
track length is long enough for a change in the angular speed to be detected, or more 
robustly, with a measurement of the arrival time of the shower at any point on the 
ground: thus a timing measurement at a single station of the SD suffices. Using this 
hybrid reconstruction method, a directional resolution of 0.5° is routinely achieved 
[40], Since only the timing information from a single SD station is needed, the hybrid 
geometry constitutes an independent and sensitive cross-check on the directional and 
core location assignments made with the SD. The precise geometry of a hybrid event 
is also the first step towards a high quality measurement of the longitudinal profile of 
a shower which, in turn, yields the energy of the primary particle and the depth of 
maximum X ma x- 

Many experimental challenges exist in fully realizing the promise of the hybrid 
technique to provide high quality measurements of shower observables. The Collabo¬ 
ration employs a series of cross-checks and measurement redundancies to understand 
the systematic uncertainties in each measurement. These cross-checks include compar¬ 
isons between SD and FD measured parameters. There are also important redundancies 
in various calibration measurements and redundancies in measurements of those atmo¬ 
spheric properties that are critical for accurate FD event reconstruction. Of particular 
importance is the aerosol content of the atmosphere. The concentration of aerosols is 
variable over timescales of hours and it can vary over the area of the Observatory. The 
concentration affects the transmission of fluorescence light from the shower to the FD 
telescopes, and the scattering of Cherenkov light into the fields of view of the tele¬ 
scopes. While the choice of the number and location of fluorescence sites around the 
surface array was driven by the desire to minimize the effects of atmospheric uncertain¬ 
ties on reconstruction, a great deal of effort is still required in atmospheric monitoring 
(see Section 9 for details). The aerosol concentration and distribution are monitored on 
timescales of 15 minutes using the two laser facilities (see Figure 1) within the detec¬ 
tion volume and by lidar systems at each FD site. In addition the directional properties 
of aerosol scattering are measured at two sites, and the wavelength dependence of the 
scattering is obtained (Section 9). 

While the threshold energy for a fully-efficient trigger for showers with zenith an¬ 
gles smaller than 60° is 3 x 10 18 eV, hybrid observations require only one SD station to 
have triggered and thus the threshold for hybrid events is significantly lower. For ex¬ 
ample, a hybrid measurement of the “elongation rate”, the energy dependence of Wnax, 
has been published for energies above 10 18 eV [21], with work underway to push the 
threshold energy lower with the HEAT fluorescence telescopes (Section 13.1). Hybrid 
showers are collected with full efficiency over the entire SD array at energies above 
10 19 eV. Around 6000 quality hybrid events are recorded per year above 10 18 eV, with 
300 per year above 10 19 eV. Quality cuts normally applied to these events include the 
requirement that the depth of shower maximum is in the field of view of one of the FD 
telescopes. 
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Figure 3: A schematic view of a surface detector station in the field, showing its main components. 


One final experimental cross-check is worth mentioning. Though the FD was not 
designed as a “stereo” instrument, a significant number of showers at the higher en¬ 
ergies are observed by more than one FD site. At 10 19 eV over 60% of FD showers 
are viewed in stereo, increasing to 90% at 3 x 10 19 eV. Stereo observations provide 
two or more independent hybrid reconstructions of the shower geometry, and of profile 
parameters such as energy and X max . This allows cross-checks of atmospheric correc¬ 
tions, and of simulations of the detector resolution. This has confirmed the statistical 
resolutions for a single site to be ~10% and 20g/cm 2 for energy and X max respectively, 
at around 10 19 eV [41], 

3. The surface detector 

3.1. Overview 

A surface detector station consists of a 3.6 m diameter water tank containing a 
sealed liner with a reflective inner surface. The liner contains 12,000 liters of ultra- 
pure water. Three 9-inch diameter Photonis XP1805/D1 photomultiplier tubes (PMTs) 
are symmetrically distributed on the surface of the liner at a distance of 1.20 m from 
the tank center axis and look downward through windows of clear polyethylene into 
the water. They record Cherenkov light produced by the passage of relativistic charged 
particles through the water. The tank height of 1.2 m makes it also sensitive to high 
energy photons, which convert to electron-positron pairs in the water volume. 

Each surface detector station is self-contained. A solar power system provides an 
average of 10 W for the PMTs and electronics package consisting of a processor, GPS 
receiver, radio transceiver and power controller. The components of a surface detector 
station are shown in Figure 3. Ref. [42] describes the surface detector in detail. 

Figure 1 shows the layout of the surface array and the FD buildings at its periphery. 

3.2. The SD station 

The tanks are made of polyethylene using the rotational molding, or rotomolding, 
process. This process, in simplified form, consists of putting a set amount of polyethy¬ 
lene resin inside a mold, then rotating the mold and heating it until the resin has melted 
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and uniformly coated the interior walls of the mold. The result is a low cost, tough, 
and uniform tank with robustness against the environmental elements. The carefully 
selected, custom compounded polyethylene resins contained additives to enhance ul¬ 
traviolet protection. The interior two-thirds of the wall thickness was compounded with 
1% carbon black to guarantee light-tightness. The outer one-third was colored beige 
to blend with the landscape. The tanks have an average wall thickness of 1.3 cm and a 
nominal weight of 530 kg. The tanks do not exceed 1.6 m in height so that they can be 
shipped over the roads within transportation regulations. 

Three hatches, located above the PMTs, provide access to the interior of the tank 
for water filling. They also provide access for installation and servicing of the interior 
parts. The hatches are covered with light- and water-tight polyethylene hatchcovers. 
For reasons of cost, durability, and ease of installation, the gaskets sealing the hatch- 
covers to the tanks may not be perfectly leak-tight, so the hatches are elevated to pre¬ 
vent accumulated water from collecting at the gasket. One hatchcover is larger than 
the other two and accommodates the electronics on its top surface. The electronics is 
protected by an aluminium dome that keeps out rain and dust. The tanks also possess 
molded-in lugs, six for lifting and four additional lugs to support the solar panel and 
antenna mast assembly. 

Electrical power is provided by two 55 Wp (watt-peak) solar panels which feed two 
12 V, 105 Ah, lead-acid, low maintenance batteries wired in series to produce a 24 V 
system. Power is expected to be available over 99% of the time. Batteries are charged 
through a commercial charge controller, which is epoxy encapsulated and has robust 
surge protection. The electronics assembly at each SD station possesses a Tank Power 
Control Board (TPCB) which monitors the power system operation. The TPCB also 
has a control function which allows the remote operator to set into hibernation any (or 
all) of the SD stations if the charge of the batteries falls below a critical level. There is 
enough reserve in the solar power system that this feature has not yet been employed. 

The batteries are accommodated in a rotationally molded polyethylene battery box. 
Since battery lifetime is reduced with increased temperature, the battery box is pro¬ 
tected from direct sunlight by installing it on the shaded side of the tank. It is also 
insulated with polystyrene foam plates to minimize high temperature excursions dur¬ 
ing the day. The box is anchored by a plate which extends under the tank. Power cables 
run through the tank interior top from feedthroughs in the large hatch to the far side of 
the tank, where they exit the tank to run to the battery box. The cables are protected 
from the point where they exit the tank to the entry of the battery box by a polyethylene 
pipe. 

The solar panels are mounted on aluminium brackets, which also support a mast. 
Antennas for radio communication and GPS reception are mounted at the top of this 
mast. The mast-and-bracket system is designed to withstand 160 km/h winds. 

The tank liners are right circular cylinders made of a flexible plastic material con¬ 
forming approximately to the inside surface of the tanks up to a height of 1.2 m. They 
enclose the water volume, provide a light-tight environment and diffusively reflect the 
Cherenkov light produced in the water volume. The liners are produced from a lami¬ 
nate composed of an opaque three-layer coextruded low density polyethylene (LDPE) 
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Figure 4: Mechanical housing for the SD PMT. Top to bottom: outer plastic housing (fez); insulating lug; 
PMT; flange; UV-transparent window. 


film bonded to a 5.6 mils (0.14 mm) thick layer of DuPont™Tyvek® 1025-BL 2 by a 
layer of titanium dioxide pigmented LDPE of 1.1 mils (28 pm) thickness. The three- 
layer coextruded film consists of a 4.5 mils (0.11 mm) thick carbon black loaded LDPE 
formulated to be opaque to single photons, sandwiched between layers of clear LDPE 
to prevent any carbon black from migrating into the water volume. Custom designing 
the laminate materials has resulted in a durable, flexible liner. 

The liner has three windows through which the PMTs look into the water volume 
from above. These windows are made of UV-transparent linear low density polyethy¬ 
lene. Each PMT is optically coupled to a window with optical GE silicone RTV-6196 
and shielded above by a light-tight plastic cover, designated as the “fez”. Figure 4 
shows the PMT enclosure. The fez has four ports, including a light-tight air vent for 
pressure relief. The other ports are for cable feedthroughs. Two foam insulation rings 
that fit inside the fez serve to prevent ice buildup near the PMT. 

Once deployed in their correct positions in the field, the tanks are filled with ultra- 
pure water produced at a water plant owned and operated by the Auger Collabora¬ 
tion. Water quality (resistivity) exceeds 15 MO cm at the output of the water plant, and 
the water is transported in clean specialized transport tanks. Tests have indicated that 
ultra-pure water does not support bacterial growth which could lead to reduced water 
clarity. Because of its high purity, the water is expected to maintain its clarity without 
significant degradation for the lifetime of the Observatory, estimated at 20 years. Occa¬ 
sional testing of the water in a sampling of detectors has shown no significant bacterial 
growth. 

3.3. The SD electronics 

To collect the Cherenkov light produced in the water volume of the detectors by 
the air showers, three PMTs view the water volume from above. The PMTs have a 
9 inch diameter photocathode and eight dynodes, with the chemical composition of the 
dynode surfaces optimized by the manufacturer to maximize linearity. Due to their 
proximity to water they are operated with a positive anode voltage, the photocathode 
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being grounded. The high voltage is provided locally from a module integrated in the 
PMT base, and is proportional to a DC control voltage provided by the slow control 
system. The PMTs are operated at a nominal gain of 2 x 10 5 , and are specified for 
operation at gains up to 10 6 . The PMTs are required to be linear within 5% up to 
50 mA anode current. To minimize the effect of the geomagnetic field on the PMTs, 
their orientation is aligned with respect to the azimuth of the Earth’s magnetic field at 
deployment. The base, including the high voltage supply, is attached to the tube by 
soldering to flying leads and is potted in GE silicone RTV-6136 to protect it from the 
high humidity present in the tank. 

Each PMT has two outputs. An AC coupled anode signal is provided. In addition, 
the signal at the last dynode is amplified and inverted by the PMT base electronics to 
provide a signal with 32 times the charge gain of the anode. No shaping of the signal 
is performed on the PMT base. 

Six identical channels of electronics are provided to digitize the anode and ampli¬ 
fied dynode signals from each of the PMTs. Each channel consists of a 5-pole Bessel 
filter with a —3dB cutoff at 20 MHz and a voltage gain of —0.5. This filter is im¬ 
plemented using a pair of Analog Devices AD8012 current feedback op-amps. The 
filtered analog signals are fed to Analog Devices AD9203 10 bit 40 MHz semi-flash 
ADCs. The ADC negative inputs are biased to —50 mV to bring the input pedestal on 
scale and allow for amplifier section offsets. The choice of filter cutoff results in 5% 
aliasing noise while preserving the time structure of the signals. The use of two 10 bit 
ADCs with a gain difference of 32 extends the dynamic range of the system to 15 bits 
with a 3% precision at the end of the overlap region. The maximum signal recorded 
before saturation corresponds to about 650 times the peak current from a vertical muon 
traversing the tank, which corresponds to the signal from a 100 EeV cosmic ray at 
about 500 meters from the shower core. 

An LED flasher is mounted in a test port of the water tank liner. The LED flasher 
incorporates two LEDs which can be pulsed independently or simultaneously and with 
variable amplitude. This allows testing of the linearity of the photomultipliers to be 
conducted remotely. 

Each SD station contains a GPS receiver with its corresponding antenna mounted 
at the top of the communications mast for event timing and communications synchro¬ 
nization. The receiver is a Motorola (OEM) Oncore UT+. This receiver outputs a 
timed one-pulse-per-second (1 PPS). The GPS 1 PPS signal is offset from the true GPS 
second by up to 50 ns, and a correction for this offset is provided periodically by the 
receiver. Event timing is determined using a custom ASIC which references the tim¬ 
ing of shower triggers to the GPS 1 PPS clock. The ASIC implements a 27 bit clock 
operating at 100 MHz. This clock is latched on the GPS 1 PPS signal at the time of 
each shower trigger. A counter operating at the 40 MHz ADC clock is also latched on 
the GPS 1 PPS clock. These data, together with the timing corrections provided by the 
GPS receiver, are used to calibrate the frequencies of the 40 MHz and 100 MHz clocks 
and to synchronize the ADC data to GPS time within 10 ns RMS. 

The digital data from the ADCs are clocked into a programmable logic device 
(PLD). In the first half of the deployment, we employed two ALTERA ACEX PLDs 
(model EP1K100QI208-2) with 16K x 36 bits additional external static RAM. In later 
stations, an Altera Cyclone FGPA replaced the two ACEX devices and external mem- 
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Figure 5: Charge spectrum obtained when a surface detector is triggered by a threefold coincidence among 
its photomultipliers (open histogram). The hatched histogram shows the spectrum when triggered on central 
vertically aligned plastic scintillators. The bin containing the peak of the scintillator triggered spectrum is 
defined as a vertical equivalent muon. The leftmost peak in the open histogram is due to low energy and 
comer-clipping muons convolved with the threefold low threshold coincidence. 


ory. The PLD implements firmware that monitors the ADC outputs for interesting 
trigger patterns, stores the data in a buffer memory, and informs the station microcon¬ 
troller when a trigger occurs. There are two local trigger levels (T1 and T2) and a 
global third level trigger, T3. Details of the local triggers are described in Section 3.5. 

The front end is interfaced to a unified board which implements the station con¬ 
troller, event timing, and slow control functions, together with a serial interface to the 
communications system. The slow control system consists of DACs and ADCs used 
to measure temperatures, voltages, and currents relevant to assessment of the operation 
of the station. 

The station controller consists of an IBM PowerPC 403 GCX-80MHz, with a 
32 MB DRAM bank to store data and executable code, and a 2 MB Flash EPROM 
for the bootstrap and storing of the OS9 operating system. The data acquisition system 
implemented on the station controller transmits the time stamps of the ~20 T2 events 
collected each second to CDAS (Central Data Acquisition System; see Section 6). 
CDAS returns T3 requests to the station within ~8 seconds of the event (including 
communications delays due to retransmission). The station controller then selects the 
T1 and T2 data corresponding to the T3 requests and builds it into an event for trans¬ 
mission to CDAS. Calibration data are included in each transmitted event. 

3.4. Calibration of the surface detector 

The Cherenkov light recorded by a surface detector is measured in units of the 
signal produced by a muon traversing the tank on a vertical trajectory (Figure 5). This 
unit is termed the Vertical Equivalent Muon (VEM). The goal of the surface detector 
calibration is to measure the value of 1 VEM in hardware units (i.e., in integrated FADC 
channels). During shower reconstruction, the signal recorded by the tanks is converted 
into units of VEM, and the total shower energy and arrival direction are fitted using a 
lateral distribution function and energy conversion based upon hybrid analysis using 
the FD. The conversion to units of VEM is done both to provide a common reference 
level between tanks and to calibrate against the detector simulations. 
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The total bandwidth available from each SD station to the CDAS is ^1200 bit/s, 
which requires that the calibration be done by the local electronics. Also, the remote¬ 
ness of the detectors implies that the calibration procedure be robust, allowing for fail¬ 
ures of individual PMTs. 

We define <2vem (denoted simply by Qvem hereafter) as the bin containing the 
peak in the charge histogram of an individual PMT response, and /yEM (denoted by 
/vem hereafter) as the bin containing the peak in the pulse height histogram. These 
quantities are used in the three main steps in the calibration procedure: 

1. Set up the end-to-end gains of each of the three PMTs to have /vem at 50 chan¬ 
nels. The choice of 50ch//yEM results in a mean gain of ~3.4 x 10 s for a mean 
/jpe/VEM of ~94 photoelectrons. 

2. To compensate for drifts, adjust the electronics level trigger by continually per¬ 
forming a local calibration to determine /vem in channels. 

3. Determine the value of (2vem to high accuracy using charge histograms, and use 
the known conversion from Qmem to 1.0 VEM to obtain a conversion from the 
integrated signal of the PMT to VEM units. 

The high voltages, and thus the gains of each of the three PMTs, are tuned to 
match a reference event rate. This tuning implies that the PMTs in the SD stations 
will not have equivalent gains, even for PMTs in the same tank. If, for example, a 
particular SD station has more detected photons per vertical muon than the average 
station, then the PMTs in this station will be operated at a lower gain than average to 
compensate. Conversely, if a PMT has a worse optical coupling than others in the same 
tank, resulting in fewer observed photons per vertical muon, the PMT will be run at a 
higher gain. 

In addition to the primary conversion from integrated channels to VEM units, the 
calibration must also be able to convert the raw FADC traces into integrated channels. 
The primary parameters needed for this are the baselines of all six FADC inputs, and 
the gain ratio between the dynode and anode. The dynode/anode, or D/A, ratio is 
determined by averaging large pulses and performing a linear time-shifted fit to obtain 
both D/A and the phase delay between the dynode and anode. This method determines 
D/A to 2%. 

The calibration parameters are determined every 60 s. The most recently deter¬ 
mined parameters are returned to CDAS with each event and stored along with the 
event data. Each event therefore contains information about the state of each SD sta¬ 
tion in the minute preceding the trigger, allowing for an accurate calibration of the data. 
Ref. [43] describes in detail the calibration method of the surface detector. 

3.5. The SD local triggers 

Several independent local trigger functions are implemented in the front-end elec¬ 
tronics: the scaler trigger, the calibration trigger, and the main shower trigger. 

The scaler trigger records pulses with a very low threshold for auxiliary physics 
purposes such as space weather. The calibration trigger collects low threshold pulses 
using a small number of bins (20), which is one bin above 0.1 /vem, thus providing 
high rate cosmic ray data. Data from the three high gain channels are stored from three 
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samples before the trigger to 20 samples after the trigger. These data are used to build 
calibration histograms such as the one shown in Figure 5, and are also used to convert 
offline the six FADC traces into VEM units. It was lowered during the deployment 
period for investigating signals from the tails of showers and to measure muon decay 
in the SD water volume [44]. 

The main trigger is the shower trigger that results in the recording of 768 samples 
(19.2 qs) of the six FADCs. It has two levels of selection. The first level, called Tl, 
has 2 independent modes. The first one is a simple threshold trigger (TH) requiring 
the coincidence of all three PMTs being above 1.75 /vem- This trigger is used to select 
large signals that are not necessarily spread in time. It is particularly effective for the 
detection of very inclined showers that have penetrated through a large atmospheric 
depth and are consequently dominantly muonic. The threshold has been adjusted to 
reduce the rate of atmospheric muon triggers from about 3 kHz to 100 Hz. The second 
Tl mode is a time-over-threshold trigger (ToT) requiring that at least 13 bins within 
a 3 qs window (120 samples) exceed a threshold of 0.2 /vem in coincidence for two 
out of the three PMTs. The ToT trigger selects sequences of small signals spread in 
time, and is thus efficient for the detection of vertical events, and more specifically 
for stations near the core of low-energy showers, or stations far from the core of high- 
energy showers. The rate of the ToT trigger depends on the shape of the muon pulse in 
the tank and averages 1.2 Hz with a rather large spread (about 1 Hz rms). The second 
trigger level, called T2, is applied to decrease the global rate of the Tl trigger down 
to about 23 Hz. While all T 1-ToT triggers are promoted T2-ToT, only Tl-TH triggers 
passing a single threshold of 3.2 /vem in coincidence for the three PMTs will pass this 
second level and become T2-TH. All T2s send their timestamp to CDAS for the global 
trigger (T3) determination. More details on the triggers can be found in Ref. [36] . 

In June 2013, the Observatory installed across the entire array two additional SD Tl 
triggers. These triggers build upon the ToT trigger in two ways, applying more sophis¬ 
ticated analysis to the FADC traces. The time-over-threshold-deconvolved (ToTd) trig¬ 
ger deconvolves the exponential tail of the diffusely reflected Cherenkov light pulses 
before applying the ToT condition. This has the effect of reducing the influence of 
muons in the trigger, since the typical signal from a muon, with fast rise time and 
~60 ns decay constant, is compressed into one or two time bins. The multiplicity-of- 
positive-steps trigger (MoPS), on the other hand, counts the number of positive-going 
signal steps in two of three PMTs within a 3 /is sliding window. The steps are required 
to be above a small FADC value (« 5 x RMS noise) and below a moderate value (« j 
vertical muon step). This reduces the influence of muons in the trigger. Both the ToTd 
and MoPS triggers also require the integrated signal to be above ss 0.5 VEM. Because 
these triggers minimize the influence of single muons, they reduce the energy threshold 
of the array, while keeping random triggers at an acceptable level. Thus they improve 
the energy reach of the SD, as well as improve the trigger efficiency for photon and 
neutrino showers. 

3.6. Operation and maintenance 

Currently more than 1660 surface detector stations are operational. Concerning the 
water Cherenkov detectors themselves, only very few failures have been detected. Only 
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a few liners were observed to leak shortly after installation. In this case, which con¬ 
stitutes the worst failure mode, the tank is emptied and brought back to the Assembly 
Building for replacement of the interior components. 

The electronics of the surface detector operates using solar power. A tank power 
control board incorporates protection circuits, signal conditioning for the monitoring 
of the solar power system, and a circuit allowing for orderly shutdown and wakeup of 
the station in the event of an extended cloudy period during winter when there could be 
inadequate solar power available to operate the station continuously. The solar power 
system has not yet experienced a dark period long enough to require shutting down the 
array for battery recharging. The most probable battery lifetime is 4.5-6 years [45], 
and batteries are changed during regular maintenance trips. 

The PMTs and electronic boards are the most critical elements of the SD stations. 
They are subject to very severe environmental conditions: temperature variations, hu¬ 
midity, salinity and dust. The failure rate of the PMTs is about 20 per year (about 
0.5%). Some high voltage (HV) module and base problems have been detected as well 
as some problems due to bad connections. All other failures except those concerning 
the PMTs (such as a broken photocathode) can be repaired in the field. It is currently 
estimated that the number of spare PMTs is sufficient for about 10 to 15 more years of 
operation. The failure rate of electronic boards is about 1% per year. Some of the prob¬ 
lems are repaired simply by reflashing the software. Most of the electronic problems 
can also be repaired on site. All the spare parts are stored in Malargiie. 

The operation of the array is monitored online and alarms are set on various pa¬ 
rameters [46]. The maintenance goal is to have no more that 20 detector stations out of 
operation at any time. Currently the achieved number is less that 10 detector stations 
out of operation. It is currently estimated that the long-term maintenance (including the 
battery change) requires about 3 field trips per week. This maintenance rate is within 
the original expectations. The maintenance is organized by the Science Operation Co¬ 
ordinator and performed by local technicians. The surface detector does not require a 
permanent presence of physicists from other Auger institutions on site. 

4. The fluorescence detector 

4.1. Overview 

The 24 telescopes of the FD overlook the SD array from four sites - Los Leones, 
Los Morados, Loma Amarilla and Coihueco [47]. Six independent telescopes are lo¬ 
cated at each FD site in a clean climate controlled building [33], an example of which 
is seen in Figure 6. A single telescope has a field of view of 30° x 30° in azimuth and 
elevation, with a minimum elevation of 1.5° above the horizon. The telescopes face 
towards the interior of the array so that the combination of the six telescopes provides 
180° coverage in azimuth. 

4.2. FD telescopes 

The details of the fluorescence detector telescope are shown in Figure 7 and an 
actual view of an installed telescope in Figure 8. The telescope design is based on 
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Figure 6: FD building at Los Leones during the day. Behind the building is a communication tower. This 
photo was taken during daytime when shutters were opened because of maintenance. 



Figure 7: Schematic view of a fluorescence telescope with a description of its main components. 


Schmidt optics because it reduces the coma aberration of large optical systems. Nitro¬ 
gen fluorescence light, emitted isotropically by an air shower, enters through a circular 
diaphragm of 1.1m radius covered with a Schott MUG-6 filter glass window. The 
filter transmission is above 50% (80%) between 310 and 390 nm (330 and 380 nm) 
in the UV range. The filter reduces the background light flux and thus improves the 
signal-to-noise ratio of the measured air shower signal. It also serves as a window over 
the aperture which keeps the space containing the telescopes and electronics clean and 
climate controlled. The shutters seen in Figure 7 are closed during daylight and also 
close automatically at night when the wind becomes too high or rain is detected. In 
addition, a fail-safe curtain is mounted behind the diaphragm to prevent daylight from 
illuminating a camera in case of a malfunction of the shutter or a failure of the Slow 
Control System. 

A simplified annular lens, which corrects spherical aberration and eliminates coma 
aberration, is mounted in the outer part of the aperture. The segmented corrector ring 
has inner and outer radii of 850 and 1100 mm, respectively. Six corrector rings were 
made from Schott BK7 glass and Borofloat was used for the rest. More details about 
the corrector ring can be found in [33, 48]. 
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Figure 8: Photo of a fluorescence telescope at Coihueco. 


The light is focused by a spherical mirror of ~3400 mm radius of curvature onto 
a spherical focal surface with radius of curvature ~1700mm. Due to its large area 
(~ 13 m 2 ), the primary mirror is segmented to reduce the cost and weight of the opti¬ 
cal system. Two alternative segmentation configurations are used: one is a tessellation 
of 36 rectangular anodized aluminium mirrors of three different sizes; the other is a 
structure of 60 hexagonal glass mirrors (of four shapes and sizes) with vacuum de¬ 
posited reflective coatings [33]. The average reflectivity of cleaned mirror segments at 
a wavelength X = 370 nm is more than 90%. 

The camera body is machined from a single aluminium block of 60 mm thick¬ 
ness, with an outer radius of curvature of 1701 mm and an inner curvature radius of 
1641 mm. The hexagonal photomultiplier tubes, model XP3062 manufactured by Pho- 
tonis, are positioned inside 40 mm diameter holes drilled through the camera block at 
the locations of the pixel centers. The pixels are arranged in a matrix of 22 rows by 20 
columns. 

The PMT boundaries are approximate hexagons with a side to side distance of 
45.6 mm. The PMTs are separated by simplified Winston cones secured to the camera 
body which collect the light to the active cathode of the photomultiplier tube. The light 
collectors serve to prevent photons from landing in the dead spaces between the PMT 
cathodes. The upper edge of the light collectors lie on the focal surface of 1743 mm 
radius. The pixel field of view defined by the upper edges corresponds to an angular 
size of 1.5°. 

All support structures and cables are distributed so as to minimize any obscuration 
in the light path. The contribution of reflection and scattering inside the optical system 
of the telescope has been measured in situ and with an airborne remotely controlled 
platform carrying an isotropic and stabilized UV light source [49]. The measured point 
spread function of the light distribution in pixels has been implemented in the software 
used in the air shower reconstruction. 

Cleaning and maintenance work has been required during years of detector oper¬ 
ation. The cleaning of the UV filter from outside has been performed several times 
because of deposited dust layers. Currently, the cleaning of all UV filters from outside 
is done three times per year. The equipment inside the building (i.e. the inner side 
of the filter, the corrector ring, the dust curtain) is cleaned less frequently, because it 
is not exposed to the outside environment. Dry and wet methods have been adopted 
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for cleaning the mirror segments and they both improve the reflectivity of mirrors. For 
telescopes, where the first cleaning took place six years after their installation, the re¬ 
flectivity increased by < 1% in the case of mirror segments in the upper rows and ~ 5% 
for mirror segments in the bottom rows, where the segments are turned slightly upward 
(see, e.g.. Figure 8). The reflectivity of a few selected mirror segments is measured 
once or twice each year and it changes less than 1% per year. 

Alignment of individual mirror segments was cross-checked with a laser on site. 
Moreover, additional methods using data measured by telescopes were used, such as 
star tracking. Central Laser Facility (CLF) and eXtreme Laser Facility (XLF) shots 
(Section 4.4.2), or a comparison of FD and SD geometry reconstruction. Only in two 
cases were a realignment of a telescope and a readjustment of camera position needed. 

4.3. FD electronics 

The FD electronics must provide a large dynamic range and strong background re¬ 
jection, while accepting any physically plausible air shower. Moreover, the electronics 
is responsible for anti-alias filtering, digitizing, and storing signals from the PMTs. 

The XP3062 photomultiplier tube is an 8-stage unit with a bialkaline photocathode 
with quantum efficiency of about 25% in the wavelength range 350 to 400 nm. The 
PMT high voltage is provided by a HV divider chain which forms a single physical 
unit together with the signal driver circuitry. This head electronics unit is soldered to 
the flying leads of the PMT [50] . 

The nominal gain for standard operation of the FD is set to 5 x 10 4 . Stabilization of 
the HV potential for large pulses, and in the presence of the low but not negligible light 
intensity of the dark sky background, is realized by employing an active network that 
uses bipolar transistors in the last three stages of the PMT. The active divider ensures 
that the gain shift due to the divider chain is less than 1% for anode currents up to about 
10 mA. The normal dark sky background on moonless nights induces an anode current 
of about 0.8 pA on each PMT. 

The head electronics for each PMT is connected to a distribution board located just 
behind the camera body. Each board serves 44 PMTs, providing high and low voltage 
and receiving the output signals. The signal is then shaped and digitized in the front- 
end electronics (FE) unit, where threshold and geometry triggers are also generated. 
Analog boards in the FE unit are designed to handle the large dynamic range required 
for air fluorescence measurements; this means a range of 15 bits and 100 ns timing. 

As the PMT data are processed, they are passed through a flexible three-stage trig¬ 
ger system implemented in firmware and software. The trigger rate of each pixel in a 
camera (first level trigger) is kept around 100 Hz by adjusting the pixel threshold level. 
The algorithm of the second level trigger searches for track segments at least five pix¬ 
els in length within a camera. The typical trigger rate per camera fluctuates between 
0.1 and 10 Hz. The third level trigger is a software algorithm designed to clean the 
air shower data stream of noise events that survive the low-level hardware triggers. It 
is optimized for the fast rejection of triggers caused by lightning, triggers caused by 
cosmic ray muon impacts on the camera and randomly triggered pixels. 

A rugged commercial computer (MPC) is associated with each telescope and serves 
to readout the event data from the front-end electronics through a FireWire interface. 
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The MPCs at each FD site are connected through a 100 Mbit Ethernet LAN switch to 
the site’s central readout computer, called an “EyePC”. This PC provides a connection 
between the communications network and the MPCs. The MPCs are diskless, thus they 
boot their Linux system through the network and store their data on the EyePCs hard 
drive. 

The events surviving all trigger levels are sent through the MPC to the EyePC, 
which builds an event from the coincident data in all telescopes at a given site and gen¬ 
erates a hybrid trigger (FD-T3) for the surface array. The event rate is about 0.012 Hz 
per site for the 24 baseline telescopes (see Sec. 12). 

4.4. FD test and calibration 

4.4.1. Laser test beams 

Throughout each night of FD operation, thousands of collimated UV laser pulses 
are directed into the atmosphere from two facilities located near the center of the SD 
(see Figure 1 and 29 in Section 9). Light scattered out of the laser pulses generates 
tracks in the same FD telescopes that also record the tracks generated by air showers. 
In contrast to high energy air showers, the direction, rate, and energy of the laser pulses 
can be preprogrammed as desired. Laser pulses can be fired at specific directions rel¬ 
ative to the ground, for example vertically, or in specific directions relative to the sky, 
for example aimed at the galactic center, Cen A or other potential sources of cosmic 
rays. An optical fiber at each laser directs a small amount of light into an adjacent SD 
station to provide hybrid laser events. 

Laser data recorded by the FD telescopes are used to measure FD performance, 
measure SD-FD time offsets, check FD pointing, and make the hourly measurements 
of aerosol optical depth vertical profiles for the atmospheric database. In addition, a 
low power roving system is available for use on a campaign basis. 

4.4.2. CLF and XLF 

Laser test pulses are provided by the CLF [51] and XLF, each of which features a 
Q-switched frequency tripled YAG laser. The spectral purity of the 355 nm light pulses 
delivered to the sky is better than 99.9%. This wavelength falls near the middle of 
the nitrogen UV fluorescence spectrum, between the two major No fluorescence bands 
of 337 nm and 357 nm. The maximum energy per pulse is nominally 7 mJ. Pulses 
of this energy produce tracks in the FD that have an approximate optical equivalence 
of lOOEeV air showers. The CLF has been in operation since 2003. The XLF was 
installed in 2008 and includes an automated calibration system that measures the beam 
energy and polarization. A major upgrade to the CLF in 2013 added a beam calibration 
system and a backscatter Raman lidar receiver. 

4.4.3. FD detector calibration 

The precise reconstruction of air shower longitudinal profiles requires the conver¬ 
sion of an ADC count to a light flux for each pixel that receives a portion of the signal 
from a shower. To this end, the absolute calibration of the detector response is essential. 
A calibrated large diameter, drum shaped light source provides an absolute, end-to-end 
calibration for each pixel of the fluorescence telescopes, with independent verification 
for some pixels by atmospheric Rayleigh scattering from vertical laser pulses. 
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Figure 9: Drum calibration source for the FD telescopes. Left: Detail of the drum (side view). Right: Sketch 
of the drum mounted on the telescope window. 

For these absolute methods, the flux of photons on the telescope aperture is inde¬ 
pendently measured. The effects of diaphragm area projection, optical filter transmit¬ 
tance, mirror reflectivity, pixel light collection efficiency and area, cathode quantum 
efficiency, PMT gain, preamp and amplifier gains, and digital conversion are all in¬ 
cluded in the end-to-end calibration procedure. 

The drum light source consists of a pulsed UV LED, emitting in a narrow band 
around 365 nm, mounted in a cylindrical shell of Teflon®, illuminating the interior of 
the 2.5 m diameter cylindrical drum, 1.4m deep. The sides and back surfaces of the 
drum are lined with Tyvek, while the front face is made of a thin sheet of Teflon, which 
transmits light diffusively. The drum is positioned at the entrance of the telescope under 
calibration, filling the aperture. Emission from the front face is Lambertian (within 
3%), and provides uniform illumination to each pixel over the full acceptance of the 
telescope. A schematic of the drum calibration source is shown in Figure 9. 

The drum light source intensity is calibrated [52] to a precision of better than 4% in 
a dark room, using a NIST calibrated photodiode as a reference. Absolute calibration 
constants are obtained from the ratio of the known pulsed flux of photons emitted by 
the drum and the corresponding ADC pulse integrals of the camera pixels. 

Periodically, a Rayleigh calibration system [47] is used as an independent check on 
the drum light source calibration. The 355 nm roving laser is positioned a few kilome¬ 
ters from the fluorescence telescope to be calibrated. The laser is directed vertically. 
The laser beam is depolarized and the pulse-to-pulse intensity monitored to a preci¬ 
sion of 5%. The scattered light, mainly from Rayleigh scattering by the molecular 
atmosphere, is then used to calibrate the fluorescence telescope. 

From the end-to-end calibration, the appropriate constants are found to be approxi¬ 
mately 4.5 photons/ADC count for each pixel. To derive a flux of photons for observed 
physics events, the integrated ADC number is multiplied by this constant and divided 
by the area of the aperture. The flux in photons per m 2 perpendicular to the arrival 
direction is thus obtained. 

An optical system for relative calibration [47] is used to monitor the long-term time 
variations in the calibration of telescopes. In each building, three light sources coupled 
to optical fibers distribute light signals to three destinations on each telescope. Signals 
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Figure 10: The relative efficiency between 280 nm and 430 nm measured for the telescope 3 at Coihueco. 
The curve is taken relative to the efficiency of the telescope at 375 nm. 


from a pulsed LED light source are brought to a Teflon diffuser at the center of the 
mirror with the light directed towards the camera. Fibers from a second xenon flash 
lamp light source end in 1 mm thick Teflon diffusers at the center of two sides of the 
camera, with the light directed at the mirror. The signals from the third source, also 
a Xenon flash lamp, are sent to ports on the sides of the entrance aperture where the 
light is directed toward reflective Tyvek targets mounted on the telescope doors, from 
which it is reflected back into the telescopes. Drifts of the temporal performance of 
pixels, mirror and aperture components can be identified by comparing measurements 
from the three light sources. The sources are also equipped with neutral density filters 
to permit linearity measurements, or with interference filters to monitor stability at 
wavelengths in the range of 330 to 410 nm. 

The relative spectral efficiencies, or multi-wavelength calibrations, of FD tele¬ 
scopes were measured using a monochromator-based drum light source with a xenon 
flasher. The measurement was done in steps of 5 nm from 270 nm to 430 nm. As de¬ 
scribed in Sec. 4.2 there are two types of mirrors and two different glass materials used 
for the corrector rings in the FD telescopes. In total eight telescopes were measured 
to have a complete coverage of the different components and a redundant measure of 
each combination. The uncertainty of these measurements is ~ 3%. An example of 
measured relative efficiency of an FD telescope is shown in Fig. 10. 

4.5. FD operation 

Beginning in 2005, the FD initially operated with 12 telescopes at two sites, Los 
Leones and Coihueco (see Figure 1). In the following years, two additional sites, Los 
Morados and Loma Amarilla, with six telescopes at each site, were brought into oper¬ 
ation. All FD telescopes are operated remotely from the central campus by shift per¬ 
sonnel. Their responsibilities include preparation of the FD for a run, making relative 
calibrations, starting and stopping runs and online checking of the quality of measured 
data [46]. 

The Slow Control System (SCS) assures a secure remote operation of the FD sys¬ 
tem. The SCS works autonomously and continuously monitors detector and weather 
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Figure 11: Conceptual schematic of the overall radio telecommunications system for the Pierre Auger Ob¬ 
servatory. 


conditions. Commands from the remote operator are accepted only if they do not vio¬ 
late safety rules that depend on the actual experimental conditions: high voltage, wind 
speed, rain, light levels inside/outside the buildings, etc. In case of external problems, 
such as power failures or communication breakdowns, the SCS performs an orderly 
shutdown, and also a subsequent start up of the fluorescence detector system if the 
conditions have changed. If parts of the SCS itself fail, the system automatically re¬ 
verts to a secure mode so that all critical system states (open shutters, high voltage on, 
etc.) are actively maintained. 

The observation of air showers via fluorescence light is possible only at night. 
Moreover, night sky brightness should be low and thus nights without a significant 
amount of direct or scattered moonlight are required. We also require that the sun be 
lower than 18° below the horizon, the moon remain below the horizon for longer than 
three hours, and that the illuminated fraction of the moon be less than 70% in the mid¬ 
dle of the night. The mean length of the dark observation period is then 17 nights each 
month. 

The on-time of the FD telescopes is currently ~15%. The value varies slightly be¬ 
tween telescopes depending on the telescope pointing and various hardware or software 
factors. The main remaining source of downtime is weather. The telescopes are not 
operated when the weather conditions become dangerous (high wind speed, rain, snow, 
etc.) and when the observed sky brightness (caused mainly by scattered moonlight) is 
too high. 
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Table 1: Performance summary for the radio data communications system for the Pierre Auger Observatory. 


Microwave backbone network 

Links 

4 

Frequency 

7 GHz 

Data rate 

24 Mbps 

Wireless LAN 

Nodes 

1660 

Frequency 

902 to 928 MHz ISM band 

Protocol 

TDMA, custom 

Subscriber Unit over-air rate 

200 kbps 

Effective payload rate 

1200 bps uplink 

Typical daily data packet loss rate 

less than 0.002% 


5. Data communications system 

The detector systems of the Observatory are deployed at widely dispersed positions 
over a very large area. To send commands and receive data from the four individual FD 
sites and from 1660 SD stations in the field, a bidirectional radio frequency telecom¬ 
munication network has been designed and deployed. The reliability of the network is 
critical to the function of the Observatory, particularly in the context of controlling the 
experiment, identifying event triggers, and collecting data recorded at each detector for 
each air shower event. 

For Auger, a custom designed system based on a two-layer hierarchy has been 
implemented. Individual surface detector stations are connected by a custom WLAN 
which is sectorized and supported by four concentration nodes. The WLAN is serviced 
by a high capacity microwave backbone network which also supports communications 
between the four fluorescence detector sites and the main campus data acquisition and 
control center. Figure 11 shows a conceptual schematic of the overall layout of the 
data communication system for the Observatory. Table 1 lists the main performance 
characteristics. 

5.1. The microwave backbone network 

The top layer of the Auger data communications system is a 34 Mbps backbone 
network made from commercial, point-to-point, dish mounted equipment operating 
in the 7 GHz band. Receivers and transmitters are mounted on five communications 
towers located at the perimeter of the array as depicted in Figure 12. The microwave 
backbone provides high speed network communications to nodes at all four FD sites 
and with the main campus. 

The microwave backbone, depicted schematically in Figure 13, consists of a set of 
paired links providing sufficient capacity to stream data to and from each of the FD 
sites as well as for collecting data from the individual surface stations. 
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Figure 12: One of the five communications towers: the one shown is deployed at the Los Leones site; see 
Figure 11. 



Figure 13: Configuration of the high capacity microwave backbone network that connects the four FD sites 
with the main campus control and data acquisition center in Malargiie. 


5.2. Wireless LAN 

The bottom layer of the Auger communication system consists of an extended 
WLAN comprising custom designed units operating in the 902 to 928 MHz ISM band. 
A point-to-point bidirectional communications link is established between each sur¬ 
face detector station and one of four communication concentrator nodes mounted on 
the four towers located at each of the fluorescence detector sites. Communication is 
achieved in a manner similar to a cellular telephone system by dividing the array into 
28 sectors, each of which contains up to 68 stations. 

Communications operations at each surface station are governed by a custom-built 
programmable Subscriber Unit used to mediate the transmission and reception of digi¬ 
tal data between the electronics board of a surface detector and the concentrator node. 
An analogous custom-built unit, called a Base Station Unit, mediates data transfer be- 
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Figure 14: A single GPS synchronized one-second TDM A frame is broken in time slots as shown. 


tween each concentrator node and the backbone network connection at each tower. 

5.3. Time division multiple access 

Transmissions to and from the stations are synchronized by GPS timing so that each 
station is assigned a particular time slot during which it is available to send and receive 
data. This Time Division Multiple Access (TDMA) scheme provides a contention 
free communication environment within the array. A one-second data frame includes 
68 uplink slots for collecting data from the array and 6 downlink slots for sending 
trigger requests and other commands to the stations. An additional 11 slots are reserved 
for network management, monitoring, and packet error control. The assignment of 
individual time slots within the one-second TDMA frame is shown in Figure 14. This 
makes available an effective bandwidth of at least 1200 bps uplink for each surface 
station and a 2400 bps for broadcast downlink. 

5.4. Error handling 

A central requirement of the Auger WLAN system for collecting data from the sur¬ 
face detector stations is very high reliability. In the wake of a typical trigger, digitized 
data from PMT traces and other detector information must be relayed promptly to the 
CDAS so that the event can be built and recorded. Data from a single event trigger will 
typically be broken into several dozens packets transmitted by each station on request, 
a process that can continue for as long as two minutes. If even a single data packet 
is missing or corrupt, the entire trace from the station is lost. A custom packetiza¬ 
tion protocol that includes Cyclic Redundancy Checking to detect transmission errors 
is used at every level. An advanced retransmit-on-error scheme, commonly called an 
Automatic Repeat Request (ARQ), is also employed. The ARQ scheme is especially 
designed to prevent data loss in the case of variable signal fluctuations, external sources 
of interference or any other episodic environmental influences. If a packet is flagged 
as missing or corrupt at the monitoring concentrator of the central network, a request 
to retransmit the packet is automatically initiated and collected via the subsequent data 
frame reserved TDMA time slots. The ARQ request will be sent once per frame and 
will be repeated so that at least six attempts are made to retrieve each missing or corrupt 
packet. 

6. Central data acquisition system 

6.1. Overview 

The CDAS has been running since March 2001. The system was designed to as¬ 
semble the triggers from the surface array detectors, to allow control of these detectors 


32 











and to organize the storage of data. It is constructed using a combination of com¬ 
mercial hardware and custom made, high level, software components. The system is 
designed to run continuously, with minimum intervention, with the full 1660 detector 
array, and can manage many more. Data from the FD are recorded separately at the FD 
locations and transferred daily to the Computer Center at Malargiie, although hybrid 
coincidences are identified online within the SD data stream. 

The primary role for the CDAS is to combine local trigger information from the 
SD stations in order to identify potential physical events generating an SD higher level 
trigger (T3). These triggers combined with the T3 from FD sites (FD T3) are used to 
generate a request for the relevant data from SD stations for these events. The CDAS 
is then used to combine and store these data to form a shower event. The CDAS also 
contains configuration and control mechanisms, the means to monitor system perfor¬ 
mance, and the tools to access and download SD monitoring, calibration, control and 
configuration data. 

Except for triggering information (see Section 6.3), the CDAS and the FD data 
acquisition systems are completely independent. The merging of FD and SD data is 
made offline during the daytime following an FD run. Data are synchronized on the 
central storage hardware after each night of observation. The newly acquired data 
within the central storage are mirrored at the primary data mirror located at the Lyon 
HEP Computer Center (France) every 3 hours; later these data can be transferred to 
secondary mirror sites. The data may then be transferred from a convenient mirror site 
to over 50 participating institutions. 

The communication between applications within the CDAS is controlled using a 
central message routine manager called the ‘Information Kernel’. This manager allows 
formatted messages to be broadcast by producer applications (applications that need to 
advertise their status), and for consumer applications (applications that need to know 
about the status of others) to receive them on demand. All data, with one exception, are 
exchanged between the CDAS applications in human readable formatted ASCII and go 
through the ‘Information Kernel’ manager. The exception is the large binary block of 
raw data coming from the SD stations. Data exchanged in raw format are calibration 
blocks and FADC traces (these comprise the event data), data from local triggers as 
well as the monitoring data. 

Since the beginning of 2013, the CDAS runs on a virtual machine using resources 
within a private Cloud installation. The Cloud is composed of 6 servers, summing up 
42 CPUs and 112 GB of RAM. CPUs are 2 GHz or faster. This scheme allows the live 
migration and automatic failover of virtual machines, to minimize the impact of critical 
failures. The disk storage system is comprised of redundant disk arrays installed in each 
server plus some standalone devices, making a total storage space near 8.5 TB, using a 
shared, replicated and distributed scheme. A Network Time Protocol (NTP) GPS clock 
is used to synchronize the system times. We have adopted the GNU/Linux Debian latest 
stable distribution as the operating system, currently v6.0r4. Only a very small fraction 
of the CPU power is used by the CDAS application processes. Most of the software 
was developed in C or C++. The whole system is installed in a Computer Center at the 
Observatory campus, with controlled temperature and redundant uninterruptible power 
supply. 
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6.2. SD data collection 

The data flow over the radio network, from individual SD stations to the central 
campus, is controlled by a dedicated application called the ‘Post Master’. The Post 
Master is the end point of the communication backbone at the Observatory Campus, 
and is designed to dispatch information extracted from the different data streams of a 
local station to the other applications of the CDAS. As its name suggests, the Post Mas¬ 
ter application is used to read the data type contained in a radio frame and to forward it 
to the proper application within the CDAS so that specific data can be handled. When 
the data received from individual SD stations are split into several radio frames, they 
are reassembled and forwarded to clients by the Post Master after all the frames have 
been received. 

The Post Master is used also to route data between the applications of the CDAS 
and the SD. Commands and configuration parameters can be transmitted, along with 
event requests, such as the level 3 trigger identified by the ‘Central Trigger’ processor. 
Software downloads over the communications link are also possible, thus enabling 
upgrades of the local DAQ software at the stations without the need to travel many 
kilometers to each one. 

Data received from each SD station belong to different data streams: 

1. Local triggers, T2: the highest priority stream, with a list of time stamps and the 
type of trigger (threshold or time over threshold), is forwarded to the ‘Central 
Trigger’ application. 

2. Shower data with its calibration data: data in this high priority stream are sent 
only when a request is received from the CDAS at an SD station. Shower events 
are split into small pieces and sent together with the T2 packets so that the avail¬ 
able bandwidth is fully used. These data are forwarded to the ‘Event Builder’ 
application. 

3. Control : this is a medium priority stream that describes the state of the detector. 
It is forwarded to the central ‘Information Kernel’ of the CDAS. 

4. Calibration and monitoring information: this is a low priority data stream. It is 
forwarded to the ‘Monitoring Recorder’ application. 

6.3. The event triggering system 

The triggering system of the Observatory fulfills two conditions. First, it detects 
showers with high efficiency across the SD array, that is more than 99% for vertical 
showers with energy above 3 x 10 18 eV. Second, it allows and identifies cross-triggers 
(hybrid events) between the FD and SD systems. Triggers from the FD use separate 
algorithms but are forwarded to the SD system to construct the hybrid data set (see 
below). 

The local DAQ system of each SD station is designed to generate low level triggers 
(T2) as described in Section 3.5. The time stamps of these triggers are sent every 
second to the CDAS. The T2 requirements are such that the average rate is always 
around 20 to 25 Hz so that at least 50% of the bandwidth is free for data transmission. 
This limitation does not reduce the global trigger efficiency (see below). At the CDAS, 
the T2s received from all stations are stored in a data block stamped according to the 
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second to which they correspond. This information also allows us to acquire the status 
of all stations of the array at each second. Once a block has existed for a time greater 
than the maximum transit time across the radio network (five seconds), it is transmitted 
to the ‘Central Trigger’ processor and discarded. 

The Central Trigger, or third level trigger T3, initiates the central data acquisition 
from the array in the three following conditions: 

1. A main trigger condition, corresponding to shower candidates, is based on both 
the time and the spatial clustering of the local triggers received from stations, 
and is described in detail below. 

2. A random trigger is generated every N minutes by selecting one of the T2s in 
an arbitrary manner, and promoting it to a T3. Currently, N = 30 but values of 3 
and 15 have also been used. The purpose of this trigger is to randomly monitor 
the FADC traces that satisfy the local trigger conditions and thus to verify the 
efficiency of the global trigger processor. 

3. A 2-fold coincidence within 1 qs of one of the two doublet stations, two couples 
of neighboring stations (10 m distant). These occur at 0.8 Hz and are scaled to 
0.0017 Hz for transmission. 

To apply the main trigger condition, the system defines concentric hexagons cen¬ 
tered in each station. The ‘Central Trigger’ processor is used to identify groups of sta¬ 
tions that are clustered in time and space as SD events. First, time clusters are sought 
by centering a window of ±25 qs on each T2. Clusters, with multiplicity of three or 
more, are then examined for spatial coincidences. 

The main trigger condition is satisfied in two modes depending on the local shower 
trigger conditions. A block diagram illustrating the logic chain and approximate rates 
of these trigger modes is shown in Figure 15. The first mode requires the coincidence 
of at least three detectors that have passed the T2-ToT trigger condition (described in 
Section 3.5) and that meet the requirement of a minimum of compactness, namely, one 
of the fired detectors must have one of the other fired detectors in the first hexagon of 
neighbors while another one is no further than the second hexagon. The second mode 
is more permissive. It requires a four-fold coincidence of any type of T2 (T2-ToT 
or T2-TH) with moderate compactness. Namely, among the four fired detectors, one 
station may be as far away as the fourth hexagon, if a station is within the first hexagon 
and another station is no further than the second hexagon. Figure 16 illustrates the 
geometric requirements of the two trigger modes. 

Once the spatial coincidence is verified, final timing criteria are imposed: each 
T2 must be within (6 + 5 n) qs of the central station, where n represents the hexagon 
number. 

Once a trigger has been identified, a T3 message requesting that all FADC trace 
information recorded within 30 qs of the central T2 is built by the ‘Central Trigger’ 
processor and forwarded to the ‘Event Builder’ and to the stations by the ‘Post Master’. 
The trigger message is also stored locally by the ‘Information Kernel’. To select which 
stations are asked for their traces the system takes each station within six hexagons of 
each of the stations whose T2 participated in the T3 construction. Additionally, the 
‘Central Trigger’ process also stores the number of T2s for each station recorded for 
monitoring purposes. 
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Figure 15: Schematics of the hierarchy of the trigger system of the Auger surface detector. 



Figure 16: Four hexagons, containing stations, are illustrated around a central surface station, for a portion 
of an ideal array. For a 3-fold coincidence, a T3 is issued if the 3 T2s are ToT, and if one of them is found 
in the first hexagon of the central station, and the other one no further than the second hexagon. A 4-fold 
coincidence applies to any kind of T2s and the additional station may be as distant as in the 4th hexagon. 
Two examples of the topology of triggers are shown: a 4-fold coincidence in which the triggered stations are 
identified by blue squares, and a 3-fold coincidence identified by red circles. 


With the arrangements described above, the total T3 trigger rate of the Observatory 
is presently of the order of 0.1 Hz and about 3 million SD events are recorded yearly. 

The DAQ system of the fluorescence detector is completely independent of the 
CDAS. Local triggers are generated at each FD site and those identified as T3 FD 
event triggers are logged by a local processor if a shower track can be found. T3 FD 
event triggers are transmitted online (within one second) from the local FD site to the 
CDAS system at the central site. The trigger information sent describes the geometry 
of the shower candidate. This includes the estimated time of arrival of the light front of 
the shower at the camera as well as the geometry of the SDP (see Section 10.2). From 
this information, the time of impact of the shower at a ground position in the region 
of the SD stations is computed and a corresponding SD event T3 is constructed. All 
FADC traces recorded within 20 p.s of the computed time are assembled as a normal 
‘SD only’ event, but with the addition of the identification of the corresponding FD T3 
trigger. 

After each night of operation, details of events recorded at the FD telescopes for 
each T3 FD event triggers are transferred to the CDAS. Data from these triggers are 
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then merged with the data collected by the SD DAQ and form the hybrid data set. A hy¬ 
brid event is therefore an ‘FD only’ event together with a special SD event that contains 
all the information from the surface stations that were in space and time coincidence 
with the FD event. 

Cross calibration of the SD and FD clocks is achieved by firing a laser into the 
sky and, at the same time, injecting a portion of the laser signal into one of the SD 
stations via an optical fiber. The time of laser pulse emission and the local time stamp 
recorded in the station are then compared. The former is reconstructed from the laser 
track recorded in the telescopes and the latter from the local trigger generated by the 
light going through the tank. 

6.4. Operation and control 

The CDAS has been designed to provide the means to monitor both its own op¬ 
eration as well as the slow control system at SD stations and various environmental 
parameters. The operation of the CDAS is monitored using a low level application that 
routinely checks that all software components are running correctly. This ‘watch dog’ 
system is used to reinitialize and relaunch any application that may have failed. Over 
the Observatory’s life, the CDAS has been operational more than 99.9% of the time. 
Most of the downtime is due to system tests, upgrades and debugging with a minimal 
impact of critical failures. 

The manager for the ‘Information Kernel’ system, used to route messages, can also 
serve as an offline monitoring tool. Its architecture is based upon one central daemon, 
the message dispatcher, to which all messages are sent automatically and transparently. 
Any application wishing to distribute its status information, or wishing to know about 
the state of other applications in the system, connects to the ‘Information Kernel’ and 
issues a monitoring request defining the class of message it wants to hear. 

The ‘message listener’ applications range from monitoring applications to system 
oriented ones, such as the message logger that is used to make all messages persistent. 
Applications have been developed that allow these messages to be browsed. Thus, the 
behavior of the system can be monitored both online and offline. The storage capacity 
is sufficient to keep the complete history of all messages exchanged in the system 
(around 3 GB per year of uncompressed ASCII files) for several years. 

The ‘Monitoring Recorder’ application of the CDAS is the core of all subsequent 
monitoring applications that will be described in next section. 

An event display program (part of which can be seen in Figure 17) has also been 
developed. This allows the selection, viewing and reconstruction of SD events that are 
stored on disk. This program, and the input/output and reconstruction libraries that 
compose it, have also been used extensively for preliminary data analysis. 

7. Monitoring 

7.1. Overview 

For the optimal scientific output of the Observatory the status of the detector array 
as well as its measured data have to be monitored. In normal operating conditions, 
shift personnel or “shifters” monitor the performance of the Auger detector systems. 
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Figure 17: Event 13357690: a typical vertical event of about 3 x 10 19 eV . Top left : The array seen from 
above with the 8 triggered stations. Top right'. The fit to the lateral distribution function (LDF) for this 
shower of zenith angle 28°. Bottom'. The FADC traces from 2 detectors at distances of 650 and 1780 meters 
from the shower core. The signal sizes are in units of VEM. 


The Auger Monitoring tool has been developed to support the shifter in judging and 
supervising the status of the detector components, the electronics, communications, 
and the data acquisition. It is also useful to study “offline” the behavior of the different 
components of our detectors and to define quality cuts on our data when necessary. 

The two hybrid detector systems, SD and FD, are operated differently and therefore 
the monitoring of their status has different requirements. The stations of the SD array 
operate continuously in semi-automated mode. Data acquisition must be monitored and 
failures of any station component (power system, PMT, communication device) must 
be detected. The data taking of the FD can only take place under specific environmental 
conditions and is organized in shifts. The sensitive cameras can only be operated on 
dark nights without strong winds or rain. This makes the operation a busy task for the 
shifters who have to judge the operating mode on the basis of the information given by 
the monitoring system. 

The technical description of the implementation is given separately for the services 
of the central server and the subcomponents. The subcomponents show the different 
ways the data flow to the central database are organized. 

7.2. Seri’er techniques 

The basis of the monitoring system is a database. We have chosen the widespread 
and publicly available MySQL database system. It includes all necessary features, e.g., 
replication and stored functions/procedures. The front end is based on a web server 
running Apache. The web site uses mainly PHP, CSS and JavaScript. An interface 
has been developed for generating visualizations. Currently the usage of Gnuplot is 
implemented via an internal system call. The second option used for generation of 
visualizations is the JPGraph package which is implemented with direct PHP calls on 
an object oriented basis with the interface defined in the inherited classes. 
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Alarms, occurrences of states that require immediate action, are first entered into 
a specified table of the database. The web front end checks this table for new entries 
and displays them on the web page. The shifter is trained to observe and acknowledge 
the alarm; when the problem is solved, he/she can declare the corresponding alarm as 
resolved. 

In addition to the detector performance and data quality monitoring, the functional¬ 
ity of the monitoring server itself has to be guaranteed. Therefore, the central services 
of the computer have to be monitored to assure that an alarm will definitely be noticed 
by the shifter. 

7.3. Surface detector 

With large ambient temperature variations, high salinity, dusty air, high humidity 
inside the tank and remoteness of access, monitoring the performance and reliability of 
the SD array is a challenge. The temperature ranges from 15°C in winter up to 60°C 
inside the electronics box in summer, with typical daily cycling of about 20°C. Inside 
the water tank the relative humidity can reach about 90%. In addition, thunderstorms 
are rather strong, with lightning that can damage electronic components. 

To monitor the whole array accurately, various sensors are installed in every sta¬ 
tion. Temperature is measured on each PMT base, on the electronics board, and on each 
battery. PMT voltage and current are also monitored, as well as solar panel voltages, 
individual battery voltage and charge current. The calibration described in Section 3.4 
is performed online every minute. A number of quantities are computed to check the 
behavior of each water Cherenkov detector: baseline values, single muon peak sig¬ 
nal, single muon average charge, dynode/anode ratio and PMT stability. The monitor¬ 
ing and calibration data are sent to the CDAS every six minutes. Dedicated software 
constantly parses the information sent to the CDAS, independently of the acquisition 
processes, and exports the data to the MySQL server. 

As an example, we show in Figure 18 the monitoring of the daily average of the 
voltages of the two batteries of a tank. While the value for Battery 1 is stable above 
12 V, it can be seen that since 22 December 2013, the value of Battery 2 has been 
decaying and is always below 12 V. When the voltage drops below 12 V for the first 
time, an alarm is triggered for shifters, so that they can make further checks on the 
history of the battery to understand the origin of the decay. Once the value becomes 
lower than 11.5 V, a more severe alarm is triggered to indicate that the battery should 
be repaired or replaced. 
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Figure 18: Evolution with time of the battery average nightly voltage of station 903 in December 2013. 
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Figure 19: Percentage of PMTs which do not satisfy the quality criteria among the functioning ones, as a 
function of time. 


A second example shows how the monitoring system is used to clean the data. 
Figure 19 displays the evolution of the fraction of PMTs that are rejected by the data 
analysis due to troubles detected by studying the monitoring data. This number itself is 
controlled within the monitoring system. The number of low-quality PMTs increased 
after 2008 because the full array of nearly 5,000 PMTs was now deployed, and the 
Observatory staff needed to carefully balance maintenance priorities. Since 2008 the 
rate of low-quality PMTs has been rather stable around 1%, except for a specific pe¬ 
riod in 2009 when communication problems did not allow reliable monitoring of the 
array. We chose to be conservative and disregard doubtful data during this period (see 
Section 12). 



Figure 20: VEM measured for 4802 PMTs. 


The monitoring tool also allows a general control of the behavior of the array. Fig¬ 
ure 20 shows the muon peak current (/vem) values for 4802 PMTs. The mean value 
of the muon peak (/vem) is at channel 45.6 with an RMS of 6.8 showing good unifor¬ 
mity of the detector response. The typical day/night variations are of the order of two 
ADC channels. This is mainly due to the sensitivity of the PMTs to temperature. The 
muon calibration is made online every minute, allowing the correction for temperature 
effects. 

The ratio Area/Peak (A/P), i.e., the ratio between the integrated charge and maxi¬ 
mum of the atmospheric muon signals recorded with the calibration trigger, is also a 
monitored quantity directly available from the local station software. It is related to the 
water transparency and the reflectivity of Cherenkov light on the inner liner of the SD 
station. These properties control the absorption length of the light and thus the signal 
decay constant. Figure 21 shows the decay of the A/P ratio of a typical station in the 
first seven years following deployment, coupled to a seasonal modulation. After 10 
years of deployment, the A/P tends to be stable. This behavior is described in detail in 
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Ref. [45]. 



Figure 21: A/P as a function of time for station 116. The dots are the average of the A/P value over one day. 

The two contributions of the T2 shower triggers (see Section 3.5) are also mon¬ 
itored. While the T2-TH mode has a mean value of 22 Hz with a low dispersion of 
less than 2%, the T2-ToT mode contributes only 1 Hz but with a much larger spread. 
Indeed, the ToT mode is directly related to the A/P, since it is by construction sensitive 
to the signal shape and thus to the characteristics of the detector. Therefore the T2-ToT 
rate also decreases with time. It has been observed that, even if the rates of the dif¬ 
ferent stations show a large initial spread across the array, most of them stabilize after 
a few years to about 1 Hz. Temperature variations also slightly affect the ToT trigger. 
Fortunately, these variations do not affect the uniformity and the stability of the data, 
since the event rate above the threshold energy of the experiment has not been affected, 
as will be shown in Section 12. 

The monitoring tool also includes performance metrics to control the overall per¬ 
formance of the Surface Detector array. One of the metrics is the number of stations 
sending a signal divided by the number of deployed tanks as a function of time, indi¬ 
cating the efficiency of data collection with the SD array, which is typically better than 
98%. 

7.4. Fluorescence detector 

The data acquisition for the FD telescopes is organized by site to insure against dis¬ 
ruption of data collection due to possible communication losses between the CDAS and 
the remote detectors. The data transport for FD monitoring is organized via a database 
internal replication mechanism. This mechanism recognizes communication problems 
and recovers submitted database changes when the connection is reestablished. This 
guarantees completeness of the dataset on the central server, even if the information is 
not immediately available online during network failures. Figure 22 shows a schematic 
layout of the databases. 

The information collected for the supervision of the FD operation is organized into 
five sections. The calibration section contains the information from the different levels 
of calibration as described in Section 4.4.3, with an example representation given in 
Figure 23. The background data section contains the information obtained from each 
30 second readout of the full camera, which is valuable as an unbiased observation of 
background. The section on DAQ and trigger shows the frequency of fired triggers that 
indicates the status of the telescopes at an advanced stage. Information from the Slow 
Control System such as rain, wind, and outside and inside temperatures is displayed 
in the fourth section. The lidars [53] monitor the atmosphere close to the telescopes. 
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Figure 22: Organization of the monitoring system databases: The single databases at each FD site are repli¬ 
cated to the database server at the central campus, while other sources like SD fill directly into the database. 
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Figure 23: Screenshot of the web interface showing a selection of FD calibration data for the six cameras in 
an FD site in a specialized view representing the PMT geometry. 


Their information helps judging the atmospheric conditions at the site, which is vital 
for the operation of the telescopes. 

The data collected in the database can be used to derive higher level quantities 
such as the on-time of the FD telescopes. This quantity is of major importance since 
it is a necessary ingredient of flux measurements. The dead time of each telescope is 
also recorded in the database. Together with the run information and other corrections 
retrieved from the database, the total on-time for each telescope can be determined 
individually. The on-time is calculated only for time intervals of ten minutes, balancing 
the statistical precision of the calculated on-time due to statistics with the information 
frequency. A program to execute the calculation runs on the database server and is used 
to fill the appropriate tables in the database continuously. The web interface displays 
the stored quantities. The on-time is available in near real time for the shifter as a 
diagnostic and figure of merit. 

7.5. Communications 

All aspects of the Auger data communications system control, operations, and per¬ 
formance housekeeping information are coordinated and reported via a central data 
concentrator node called NetMon, which also serves as the relay for all data transferred 
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between the detectors and the array control center. NetMon enables regular monitoring 
of the system performance, including critical details on the status of data links, the sta¬ 
tus of sectors and packet error rates. All of this information is integrated into the main 
monitoring system so that the on site operator can be made aware of any difficulties that 
may arise in near real time. For example. Figure 24 shows a monitoring event display 
of the radio signal strength (in dB) of the uplink receiver as reported to the operator 
for each surface detector station in the array. Long term performance benchmarks for 
the data communications system are also recorded and monitored. Figure 25 shows the 
daily number of ARQs over the course of four months of data collected during 2010 
and 2011. Here, the ARQ rate serves as a global diagnostic for the overall health of the 
system indicating the extent to which packets are missing or corrupt. Shown are the 
rates for single ARQ generating packet errors and also the rates for “ARQ 7” errors cor¬ 
responding to those packets for which at least six retransmission attempts were made 
before abandoning the packet; the ARQ 7 rate thus represents the rate at which data 
are irretrievably lost. We had for a period in 2009 a large rate of error rate leading to 
event loss. It has been fixed and since then, the typical loss rate has been approximately 
1000 to 2000 packets out of over 140 million packets per day, corresponding to a data 
loss rate of less than 0.002%. The data from a typical event involving several stations 
corresponds to approximately 1000 packets. 

Therefore, the overall event loss rate due to communication failures is less than 2%. 



Figure 24: Example of near real time performance monitoring of the Auger data communications system 
showing a map of the radio signal strengths (in dB) for each detector in the array. 


8. Data processing and Off line framework 

The Off line software of the Pierre Auger Observatory provides both an implemen¬ 
tation of simulation and reconstruction algorithms, discussed later, and an infrastruc¬ 
ture to support the development of such algorithms leading ultimately to a complete 
simulation, reconstruction and analysis pipeline. Indeed, when the Off line code was 
originally devised, the only existing systems were the SD and FD. It has since been ex- 
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Figure 25: Example of benchmark long term performance monitoring of the Auger data communications 
system showing the daily rate of ARQs over a period of 12 months from June 2010 to May 2011. Upper 
points indicate the rate of any ARQ; almost all of these result in successful retrieval of a lost or corrupt data 
packet. Lower points (ARQ 7) correspond to packets which were lost after at least six attempts to retransmit. 


tended to handle the radio and AMIGA extensions without requiring dramatic frame¬ 
work changes. The software has been designed to accommodate contributions from a 
large number of physicists developing C++ applications over the long lifetime of the 
experiment. The essential features include a “plug-in” mechanism for physics algo¬ 
rithms together with machinery which assists users in retrieving event data and detec¬ 
tor conditions from various data sources, as well as a reasonably straightforward way 
of configuring the abundance of different applications and logging all configuration 
data. A detailed description of the Off line software design, including some example 
applications, is available in [54], 

The overall organization of the Off line framework is depicted in Figure 26. A col¬ 
lection of processing modules is assembled and sequenced through instructions con¬ 
tained in an XML file [55] or in a Python [56] script. An event data model allows 
modules to relay data to one another, accumulates all simulation and reconstruction 
information, and converts between various formats used to store data on file. Finally, 
a detector description provides a gateway to data on the detector conditions, including 
calibration constants and atmospheric properties as a function of time. 

8.1. Physics modules 

Most simulation and reconstruction tasks can be factorized into sequences of pro¬ 
cessing steps which can simply be pipelined. Physicists prepare processing algorithms 
in modules, which they register with the Off line framework via a one line macro. This 
modular design allows collaborators to exchange code, compare algorithms and build 
up a variety of applications by combining modules in various sequences. Run time 
control over module sequences is implemented with a Run Controller, which invokes 
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Figure 26: General structure of the Off line framework. Simulation and reconstruction tasks are encased in 
modules. Each module is able to read information from the detector description and/or the event, process 
the information, and write the results back into the event under command of a Run Controller. A Central 
Config object is responsible for handing modules and framework components their configuration data and 
for tracking provenance. 


<sequenceFile> 

<loop numTimes="unbounded"> 
<module> SimulatedShowerReade 
<loop numTimes="10"> 

<module> EventGenerator < 

<module> TankSimulator < 

<module> TriggerSimulator < 

<module> EventExporter < 

</loop> 

</loop> 

</sequenceFile> 


r </module> 

/module> 

/module> 

/module> 

/module> 


Figure 27: A simplified example in which an XML file sets a sequence of modules to conduct a simulation 
of the surface array. <loop> and <module> tags are interpreted by the run controller, which invokes the 
modules in the proper sequence. In this example, simulated showers are read from a file, and each shower is 
thrown onto the array in 10 random positions by an EventGenerator. Subsequent modules simulate the 
response of the surface detectors and trigger, and export the simulated data to file. Note that XML naturally 
accommodates common sequencing requirements such as nested loops. 


the various processing steps within the modules according to a set of user provided 
instructions. We devised an XML-based language as one option for specifying se¬ 
quencing instructions; this approach has proved sufficiently flexible for the majority 
of our applications, and it is quite simple to use. An example of the structure of a 
sequence file is shown in Figure 27. 

8.2. Data access 

The Off line framework includes two parallel hierarchies for accessing data; the 
detector description for retrieving data on conditions, including detector geometry, cal¬ 
ibration constants, and atmospheric conditions, and an event data model for reading 
and writing information that changes for each event. 

The detector description provides a unified interface from which module authors 
can retrieve conditions data. Data requests are passed by this interface to a back end 
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Figure 28: Machinery of the detector description. The user interface (left) comprises a hierarchy of objects 
describing the various components of the Observatory. These objects relay requests for data to a registry of 
managers (right) which handle multiple data sources and formats. 


comprising a registry of so-called managers, each of which is capable of extracting a 
particular sort of information from a given data source. The manager mechanism al¬ 
lows for a simple interface to a potentially complex collection of different data sources 
and formats. The general structure of the detector description machinery is illustrated 
in Figure 28. 

The event data model contains the raw, calibrated, reconstructed and Monte Carlo 
information and serves as the backbone for communication between modules. The 
event is instrumented with a protocol allowing modules to discover its constituents at 
any point in processing, and thereby determine whether the input data required to carry 
out the desired processing are available. 

The transient (in memory) and persistent (on disk) event models are decoupled in 
order to avoid locking to a single provider solution for serialization, the process by 
which C++ objects are converted to a form that can be written to disk. When a request 
is made to write event contents to file, the data are transferred from the transient event 
through a file interface to the persistent event, which is instrumented with serialization 
machinery. We currently use the input/output portion of the ROOT [57] toolkit to 
implement serialization. Various file formats are interpreted using the file interface, 
including numerous raw event and monitoring formats as well as the different formats 
employed by the AIRES [58], CORSIKA [59], CONEX [60] and SENECA [61] air 
shower simulation packages. 

8.3. Configuration 

The Off line framework includes a system to organize and track data used to con¬ 
figure the software for different applications as well as parameters used in the physics 
modules. A Central Config configuration tool (Figure 26) points modules and frame¬ 
work components to the location of their configuration data, and creates Xerces-based [62] 
XML parsers to assist in reading information from these locations. We have wrapped 
Xerces with our own interface which provides ease of use at the cost of somewhat re¬ 
duced flexibility, and which also adds functionality such as automatic units conversion 
and casting to various types, including commonly used containers. 

The Central Config keeps track of all configuration data accessed during a run 
and stores them in an XML log file, which can subsequently be used to reproduce a 
run with an identical configuration. This allows collaborators to easily exchange and 
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use configuration data for result comparisons. The logging mechanism is also used to 
record the versions of modules and external libraries which are used for each run. 

Syntax and content checking of the configuration files is afforded through W3C 
XML Schema [63] standard validation. Schema validation is used not only for internal 
configuration prepared by framework developers, but also to check the contents of 
physics module configuration files. This approach reduces the amount of code users 
and developers must prepare and supports very robust checking. 

8.4. Utilities, testing and quality control, and build system 

The Off line framework is complemented by a collection of utilities, including an 
XML parser, an error logger and various mathematics and physics services. We have 
also developed a novel geometry package which allows the manipulation of abstract ge¬ 
ometrical objects independent of coordinate system choice. This is particularly helpful 
for our applications since the Observatory comprises many instruments spread over a 
large area and oriented in different directions, and hence possesses no naturally pre¬ 
ferred coordinate system. Furthermore, the geometry package supports conversions to 
and from geodetic coordinates based on a reference ellipsoid. 

As in many large scale software development efforts, each low level component of 
the framework is verified with a small test program, known as a unit test. We have 
adopted the CppUnit [64] testing framework as an aid in implementing these tests. In 
addition to unit tests, a set of higher level acceptance tests has been developed which 
is used to verify that complete applications continue to function as expected, within 
some tolerance, during ongoing development. We employ a BuildBot system [65] 
to automatically compile the Off line software, run the unit and acceptance tests, and 
inform developers of any problems each time the code is modified. 

The Off line build system is based on the CMake [66] cross-platform build tool, 
which has proven sufficient for this project. In order to ease installation of Off line 
and its various external dependencies, we have prepared a tool known as APE (Auger 
Package and Environment) [67], APE is a python-based dependency resolution engine, 
which downloads the external packages required for a particular application, builds 
them in whatever native build system applies for each package, and sets up the user’s 
environment accordingly. APE is freely available, and has been adopted by other ex¬ 
periments, including HAWC, NA61/SHINE and JEM-EUSO. 

8.5. O ffline summary 

At the time of writing, the Off line software comprises over 350,000 lines of code, 
corresponding to some 95 person years of effort, according to the Constructive Cost 
Model [68]. The framework has been enhanced for simulation and reconstruction of 
the Observatory extensions discussed in Sections 13.1 and 13.2 and for the radio re¬ 
search program described in Section 16.1, for which substantial additions to the Off line 
functionality were developed [69]. The code is available under an open source BSD 
license upon request. Other experiments have adopted portions of the Off line code, in¬ 
cluding Tunka-Rex [70], HAWC [71], JEM-EUSO [72], CODALEMA [73] and NA61- 
/SHINE [74, 75], 
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9. Atmospheric monitoring 

The Observatory makes use of the atmosphere as a giant calorimeter. This moti¬ 
vated the selection of a site with generally good viewing conditions and the implemen¬ 
tation of an extensive program to monitor the troposphere above the site. A detailed 
knowledge of the atmosphere is required for the accurate reconstruction of air show¬ 
ers observed by the FD [76-79] and for the accurate estimation of the exposure of the 
detectors [37]. 

The atmospheric state variables, including temperature, pressure and humidity, are 
needed to assess the longitudinal development of extensive air showers [77, 80] as well 
as the amount of the isotropically emitted fluorescence light induced by the air show¬ 
ers [81-84]. The SD observations are altered by different atmospheric conditions [85]. 
Varying air densities close to the ground modify the Moliere radius affecting the lat¬ 
eral distribution of the electromagnetic component of the extensive air shower (EAS). 
Varying air pressure affects the trigger probability and the rate of events detected above 
a fixed energy. Furthermore, the atmospheric state variables are used to determine the 
Rayleigh (pure molecular) scattering of the fluorescence and Cherenkov light. Installa¬ 
tions for recording local conditions of the state variables are described in Section 9.1. 


FD Loma Amarilla 
Lidar 

IR Camera 



FD Los Leones 
Lidar, HAM, FRAM 
IR Camera 
Weather Station 


10 km 


Figure 29: Schematic overview of the atmospheric monitoring devices installed at the Pierre Auger Obser¬ 
vatory. At each FD site, there is a lidar station, a ground-based weather station, and an infrared camera 
for cloud cover detection. In addition, there are devices for measuring the Aerosol Phase Function (APF) 
at FD Coihueco and Los Morados, a Horizontal Attenuation Monitor (HAM) at FD Los Leones, and a 
ph(F)otometric Robotic Atmospheric Monitor (FRAM) also at Los Leones. A steerable backscatter elastic 
lidar system is installed at each of the 4 FD sites to measure aerosols and the positions of clouds near each 
site. At central positions within the surface detector array, two laser facilities are installed (CLF and XLF). 
These instruments, together with the FD, are used to measure T aer (/z) in the line of sight of each FD telescope 
4 times per hour. In April of 2013 the CLF was upgraded with a Raman lidar receiver. Near the western 
boundary of the array, the Balloon Launching Station (BLS) was assembled together with a weather station 
as a base unit for an electric field meter. From this launch station, the weather balloons were typically earned 
across the entire array by westerly winds. 

Aerosols and clouds represent the most dynamic monitoring and calibration chal- 
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lenges at the Observatory. The optical transmission properties of the atmosphere, in¬ 
cluding the vertical aerosol optical depth profile T aer (/t), have to be measured across 
the Observatory during FD data taking. In the air shower reconstruction, the atmo¬ 
spheric transmission between the FD and an air shower must be taken into account to 
properly reconstruct the light generated along the shower axis from the light recorded 
at the telescope(s) [76, 79], Moreover, Cherenkov light induced by the air showers is 
also detected with the FD and needs to be reconstructed as a function of atmospheric 
conditions at the time of the event. Installations dedicated for determining the optical 
scattering and absorption behavior of the atmosphere in the field of view are described 
in Section 9.2 and those for identifying and determining clouds and the general extinc¬ 
tion above the Observatory in Section 9.3. 

An extensive system of atmospheric monitoring devices has been installed (Fig¬ 
ure 29). The types of measurements possible with these instruments are listed in Ta¬ 
ble 2. 


Table 2: Atmospheric measurements performed and the instruments that are used. 


Category 

Variable 

Frequency 

Instrument(s) 

State 

At ground: 

Pressure, Temp., 

Wind, Humidity 

5 min 

Weather Stations 


Profile: Pressure, 

Temp., Humidity 

3 hours 

GDAS" 

Aerosols 

Vert. Optical Depth (z) 

hourly 

CLF, XLF + FD 


Phase Function 

hourly 

2 APF units 


Angstrom Coefficent 

hourly 

FRAM (HAM) 

Clouds 

Presence in FD pixels 

15 min 

4 Cloud Cameras 


Behind FD sites 

15 min 

4 lidar stations 


Along select tracks 

avg. 1/night 

FRAM, lidar 


Above CLF/XLF 

hourly 

CLF, XLF + FD 


“atmospheric model developed at the National Centers for Environmental Prediction, operated by NOAA; 
provided via READY - Real time Environmental Applications and Display sYstem. 


9.1. Installations for atmospheric state variables 

9.1.1. Ground-based weather stations 

The Auger Collaboration operates several weather stations, as indicated in Fig¬ 
ure 29. Some of these stations are used for operational control of the nearby installa¬ 
tions. The data from the weather stations at each FD site and at the CLF additionally 
serve as atmospheric ground information in several parts of the air shower reconstruc¬ 
tion. Typically, those data are transferred via the central campus in Malargiie, processed 
and stored in our databases for atmospheric monitoring information (cf. Section 9.5) 
within a couple of days. 

The weather stations are commercial products 3 equipped with temperature, pres- 


3 Campbell Scientific, http : //www . Campbellsci . com 
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sure, humidity, and wind speed sensors recording data every 5 minutes. The stations 
at FD buildings Los Leones and Coihueco and at the laser facilities are additionally 
equipped with a sensor for wind direction. Formerly at the Balloon Launching Station 
(BLS) site and now at the AERA site (cf. Section 16.1), the weather station serves as a 
base unit for an electric field meter. The values of the electric field are recorded every 
second for lightning and thunderstorm detection which is particularly important for the 
radio detection technique. 

9.1.2. Balloon Launching Station 

For a proper reconstruction of the fluorescence telescope signals, not only are 
ground-based atmospheric data needed, but also atmospheric profiles of the state vari¬ 
ables temperature, pressure, and humidity up to about 20 to 25 km a.s.l. [77, 80-82, 86]. 
From these directly measured values, the derived quantities air density and atmospheric 
depth are calculated. The program of launching meteorological radiosondes attached to 
helium filled weather balloons was started at the Observatory site in August 2002. After 
331 successfully measured profiles, the routine operation was terminated in December 
2010 [78] and then replaced by the meanwhile validated GDAS data. During the first 
years, campaigns of about three weeks with an average of nine launches per campaign 
were done roughly three times a year. The starts of the soundings were usually placed 
at some FD buildings, mostly at Los Leones and Coihueco. In 2005, a dedicated BLS, 
cf. Figure 29, was installed at a suitable position to optimally cover the large area above 
the surface detector array and in the field of view of the FD telescopes by the weather 
balloons. From this fully equipped station, more regular launches could be managed, 
in particular during the night. Between July 2005 and March 2009, roughly one launch 
was performed about every five days. Between 2009 and 2011, the program was part 
of the rapid atmospheric monitoring system of the Pierre Auger Observatory (see Sec¬ 
tion 9.4). A radiosonde launch was triggered shortly after the detection of particularly 
interesting air showers such as very energetic events. Since 2011, the BLS is used for 
dedicated measurement campaigns. 

The radiosondes and the receiver station are standard meteorological products 4 pro¬ 
viding data on the temperature, pressure, humidity, and GPS position including alti¬ 
tude. Typically, a set of measurement values is recorded every 2 to 8 seconds. The 
upper limit of the profile is given by the height of the burst of the weather balloon, 
typically at about 23 km, with a few balloons reaching a maximum altitude of 27 km. 

Based on the locally measured atmospheric profiles, monthly models of atmo¬ 
spheric conditions at the Pampa Amarilla were derived in December 2008 [78, 79]. 
The monthly models are also compiled for application in air shower simulations like 
CORSIKA [59], Finally, these measurements were used to validate the utility of data 
from the Global Data Assimilation System (GDAS) for the purpose of air shower re¬ 
construction at the Pierre Auger Observatory [77]. GDAS is the result of atmospheric 
computer analyses and forcasts which are run serveral times per day and are based 
on meteorological measurements from all around the world. The data are available 
in 3-hourly, global, 1° latitude-longitude (360° by 180°) datasets. The position of the 


4 http : //www . graw . de 
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Figure 30: Difference of locally measured radiosonde data to according GDAS profiles versus height for all 
radio soundings performed in 2009 and 2010 (black dots). In addition, the difference of the same radiosonde 
data to according monthly mean profiles (red squares) are plotted. The monthly mean profiles used for this 
comparison are those developed end 2008, representing a totally independent dataset of local records than 
the plotted radiosonde data [77]. 


chosen GDAS grid point is marked in Fig. 29. Each GDAS dataset consists of surface 
data together with data for 23 constant pressure levels reaching up to about 26 km. For 
the purpose of the Auger Observatory, maily the information for temperature, pres¬ 
sure and humidity are used. These GDAS data have been compared with according 
local radio soundings and the records from the ground-based weather stations. The 
agreement of the locally recorded data with the GDAS data for the given grid point is 
well enough for the application in air shower physics, c.f. Fig. 30 . The variation for 
temperature is below ±1 K for altitudes between ground and about 20 km a.s.l., for 
pressure, the variation is below 0.5 hPa at worst, but for most of the altitude range well 
below 0.3 hPa. Even the vapor pressure is well below 0.3 hPa except for the data point 
closest to ground where the difference goes up to 1.0 hPa. After validating the utility 
of the GDAS data for the site of the Pierre Auger Observatory [77], advanced monthly 
models were compiled from GDAS data for this location for air shower simulations 
and reconstructions [78, 87], 

9.2. Installations for atmospheric transmission 

The transition of fluorescence light, incduced by extensive air showers, in the at¬ 
mosphere is reduced by absorption and scattering of the UV photons. For a correct 
reconstruction of the energy deposit of an extensive air shower in the atmosphere, the 
attenuation of the fluorescence photons has to be know for the time of the air shower. 

The attenuation of light in the wavelengths range of interest here, is dominated 
by scattering rather than by absorption. The scattering of photons in air can be de¬ 
scribed analytically from molecular scattering theory. Once the vertical profiles of at¬ 
mospheric temperature, pressure, and humidity are known, the molecular transmission 
factor T mo i(X,s) is a function of the total wavelength-dependent Rayleigh scattering 
cross section along the line of sight ,v. The scattering of photons by aerosols can be de¬ 
scribed by Mie scattering theory, but for real conditions with strongly varying shapes 
and amounts of aerosols, local measurements are needed. The knowledge of the aerosol 
transmission factor T aer (X,s) depends on frequent field measurements of the vertical 
aerosol optical depth X aer (h), the integral of the aerosol extinction a aer (z) from the 
ground to a point at altitude h observed at an given elevation angle. 


51 









Aerosol optical depth @ 3.5 km 



: "i'A 

u 








: -f+ 
4+. 




• Los Leones < 

x a > = ( 

.040 : 

- -? 

t 

% 


■ Los Morados < 

x a > = ( 

.042 : 

♦ 


A 

+ 


A Coihuoci 


-x a > - 1 

.038 ■ 



'♦'-p- Quality cut for data 
^ T a > 90 % 




Clear nigh 

s 

•% 



Dirty night 


0. 

2 0. 

4 0. 

6 0. 

8 0.1 0.12 0.14 0. 

!'** 

*** 


x a (3.5 km,355 nm) 


Figure 31: Left: The CLF. Right: Vertical aerosol optical depth at 3.5 km above the observatory obtained 
from FD measurements of CLF vertical laser shots recorded between January 2004 and December 2010. 
Data from 3 fluorescence stations are shown and the quality cut between clear and dirty nights is indicated 
[89], 


9.2.1. Aerosol optical depth profiles and clouds: CLF and XLF 

Laser tracks from the CLF and XLF (Section 4.4.2) are recorded by the 4 FD sites. 
They are used to obtain hourly measurements of the aerosol optical depth profiles [88] 
that are used in the reconstruction of each FD air shower event. Sets of 50 vertical 
shots are measured every 15 minutes by the FD telescopes throughout each night. The 
polarization of the CLF beam is randomized so that the amount of light scattered out of 
the beam is azimuthally symmetric about the beam axis. The CLF data sample began 
in 2004 and that for the XLF in 2009. These samples contain more than 1.5 million 
laser shots corresponding to more than 4 million tracks in the FD telescopes. To obtain 
the aerosol optical depth profile for each FD site, two techniques are used: the 50-shot 
averages are compared to averages collected under clear conditions (Data Normalized 
Analysis [88]), and to simulations generated with different aerosol attenuation condi¬ 
tions (Laser Simulation Analysis [88]). Data from the aerosol phase function monitors 
(next paragraph) can be used to cross check the clear reference periods. The technique 
is independent of the absolute photometric calibrations of the lasers and of the FD, and 
provides essential information in the reconstruction of hybrid and FD data. In figure 31, 
data from the vertical aerosol optical depth at 3.5 km above the fluorescence telescopes 
are shown. 

The analysis of the light profile of laser shots, as detected by the FD, also provides 
measurements of the heights of clouds directly above the CLF or XLF [88] . 

9.2.2. Aerosol Phase Function monitors 

The atmospheric scattering of both fluorescence and Cherenkov light from exten¬ 
sive air showers occurs over a range of scattering angles. The scattering angular dis¬ 
tribution (phase function) can be estimated analytically for the atmospheric molecular 
component. For the aerosol component this function depends on the size and shape 
of the aerosols [76]. The scattering function is characterized in-situ to implement a 
suitable parametrization of the scattering behavior by the air shower reconstruction. 

The Aerosol Phase Function monitors use a collimated Xenon flash lamp to direct 
light between 350 and 390 nm horizontally across the FD field of view at Coihueco 
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and Los Morados. The FD measures the light as a function of scattering angle (30° to 
150°). An analysis including data over several years revealed that a parameterization 
as 



(1) 


describes the aerosol scattering at the Observatory site reasonably well. The first term 
is a Henyey-Greenstein function [90], corresponding to forward scattering, and the 
second term accounts for the peak at large 6 typically found in the angular distribution 
of aerosol-scattered light [91]. The quantity g describes the asymmetry of scattering, 
and / determines the relative strength of the forward and backward scattering peaks. 

An average value of the phase function asymmetry parameter# of 0.56±0.10 is used in 
the Auger air shower analysis for nights with Mie scattering. For clear nights without 
any aerosols, g is set to zero. To also allow for very small aerosol content during almost 
clear nights, causing only small asymmetries in the phase function, an uncertainty of 
0.2 is estimated and attached to the value of g equals zero. 

9.2.3. Horizontal Attenuation Monitor 

The wavelength dependence of the aerosol attenuation is modeled by a falling 
power law with an exponential parameter y. The value of y varies inversely with the 
typical size of the aerosol particles. In the limit of clean air, y « 4. At the Observatory, 
y is obtained by the Horizontal Attenuation Monitor (HAM). The HAM consists of a 
high intensity discharge lamp installed close to the FD site at Coihueco. Light from 
this lamp is measured by a filtered CCD camera at the Los Leones FD site, about 45 km 
away [76], Total horizontal atmospheric attenuation is measured over this path at five 
wavelengths between 350 and 550 nm. The data indicate that the atmosphere of the 
Observatory is quite desert-like with weak wavelength dependence. An average y of 
about 0.7 with an RMS of 0.5 is used as a parameter in the air shower reconstruction. 

9.3. Installations for clouds and extinction 
9.3.1. Cloud identification 

The presence of clouds can alter the observed optical signatures of an EAS and 
reduce the aperture of the FD. Clouds can attenuate or block light from an air shower, 
producing a dip in the longitudinal profile observed by the FD. Conversely, if a shower 
passes through a cloud layer, the cloud can enhance the scattering of the intense Cherenkov 
light beam, producing a bump [92], 

The Observatory uses measurements from infrared cameras and lidar systems and 
FD measurements of CLF and XLF tracks to detect clouds. A cloud is warmer than 
the surrounding atmosphere and produces an infrared signal that depends on the cloud 
temperature and emissivity (or optical depth). An infrared camera (Raytheon Con- 
trolIR 2000B) mounted on a pan-and-tilt scanning platform, operates at each FD build¬ 
ing. The cameras are sensitive in the 7 to 13 jum band, appropriate for the peak of the 
blackbody radiation from thick clouds, but unfortunately also sensitive to water emis¬ 
sion bands even in clear sky. Every 5 minutes each camera scans the held of view 
(FOV) of the telescopes, and every 15 minutes the entire hemisphere is imaged. The 
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raw FOV images are converted into a binary image (cloudy/clear) after a series of im¬ 
age processing steps designed to remove camera artefacts and account for the expected 
elevation angle dependence of the clear sky intensity [93], These data are then mapped 
onto FD pixel directions to indicate the presence or absence of clouds in each FD pixel. 
At the time of writing, the Raytheon IR cameras are being replaced by Xenics Gobi 384 
radiometric IR cameras to improve image processing and reduce cable maintenance. 

Data from the Geostationary Operational Environmental Satellites (GOES) are also 
being analyzed [94]. GOES instruments provide radiance data in one visible and four 
infrared bands from which brightness temperatures are derived. The GOES-12 imager 
instrument covers the area of the Pierre Auger Observatory every 30 minutes. Cloud 
probability maps with a grid of 2.4 km by 5.5 km pixel size are derived for the area of 
the Observatory. Cross checks of cloud identification as derived from GOES measure¬ 
ments for the pixels viewing the CLF and as derived from FD measurements of CLF 
vertical tracks show a reliable correlation. 

9.3.2. Clouds and aerosols: FD lidar stations 

Four elastic lidars, installed next to each FD station, are used to measure cloud 
cover, cloud height and aerosols [53, 95]. Each lidar has a Nd:YLF laser that produces 
0.1 mJ pulses at a wavelength of 351 nm. Three 80 cm mirrors and a 20 cm mirror 
collect the backscattered light which is measured with Hamamatsu R7400U photo¬ 
multipliers. A UG-1 optical filter reduces background light. The lasers are operated 
remotely at a repetition rate of 333 Hz. Thousands of pulses are averaged by analog 
and photon counting readout systems. The two traces from these parallel readout paths 
are then combined to cover from 200 m up to 25-30 km. 

The lidars are steerable and perform discrete and continuous scanning patterns au¬ 
tomatically during the FD operation. To avoid interference with FD data collection, 
most of the scanning shots are aimed outside the FD field of view. Two exceptions 
are horizontal shots fired in the direction of the CLF to measure ground level aerosol 
horizontal attenuation length and shoot-the-shower scans to probe the detector shower 
plane shortly after especially interesting cosmic ray candidates have been observed. 

Long term measurements from these lidars find a mean aerosol extinction length at 
ground level of 0.028 km, which indicates that the atmosphere at the site is quite clear. 
Furthermore, it can be derived that about 62% of the FD data taking time is quite clear 
with a mean cloud cover of less than 20%. 

9.3.3. FRAM 

The ph(F)otometric Robotic Atmospheric Monitor 
(FRAM) is an optical telescope (0.3 m diameter mirror) that measures starlight to de¬ 
termine the wavelength dependence of Rayleigh and Mie scattering. It is also used 
to make automatic observations of light curves of optical transients associated with 
gamma ray bursts [96] . 

FRAM was installed at the site of the Los Leones fluorescence detector in 2005. 
Since the end of 2009, it has been part of the rapid atmospheric monitoring program, 
cf. Section 9.4. Because the FRAM is a passive instrument, it can operate in the field 
of view of the FD. A photometric observation of selected standard (i.e., non-variable) 
stars has been supplemented with a photometric analysis of CCD images since 2011 
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[79, 97]. A wide field camera is used to measure the atmospheric extinction along 
the shower detector plane. Its field of view is 240’ (4°) in azimuth (aligned with right 
ascension) and 160’ (2.67°) in elevation (aligned with declination). This instrument is 
a Finger Lake Instrumentation (FLI) MaxCam CM8 with Carl Zeiss Sonnar 200 mm 
//2.8 telephoto lens. A second, narrow field camera is used to calibrate the images of 
the wide field camera. Before June 2010, this camera was also a FLI instrument, but 
was exchanged for a Moravian Instruments CCD camera G2. 

9.4. Rapid monitoring 

In 2009, the atmospheric monitoring program was upgraded [79] to probe the 
shower FD detector plane with the FRAM and lidar systems a few minutes after the 
FD detected any extremely high energy EAS or an EAS with an unusual longitudinal 
profile. All atmospheric subsystems involved use individual and modifiable trigger set¬ 
tings to identify these kind of FD detected air showers. The motivation was to check 
for clouds or aerosol layers that might distort the observed profile. Between March 
2009 and December 2010, a weather balloon was also released from the BLS site to 
measure the pressure, temperature and humidity profiles within a few hours of the EAS 
detection above the Auger array with an energy above 10 19 eV. During this period, 
100 FD events were triggered for a weather balloon launch. Some of the triggers were 
received while a radiosonde was already in flight, due to the tendency of high-quality, 
high-energy observations to cluster during very clear, cloudless nights. Thus, 62 trig¬ 
gers were covered by 52 flights. The remaining triggers were lost due to technical 
issues such as hardware failures at the BLS, problems with the transmission of the text 
message to the technician who needed to launch the weather balloon, or other failures 
in the radiosonde flights. 

9.5. Atmospheric databases 

The atmospheric monitoring data are organized into MySQL databases that are 
accessed by the Auger offline analysis package for air shower reconstruction, condition 
assessment, and aperture estimates. 

The cloud information includes the IR cloud camera distributions mapped onto the 
FD pixels and the hourly measurement of the fraction of the sky covered by cloud 
measured by the lidars. An hourly cloud coverage below 20% is required for hybrid 
events to be used in the analysis of the mass composition and energy spectrum of the 
cosmic rays [21,38], 

The aerosol optical database contains, for each FD site, hourly T aer (/i) profiles in 
200 m steps and derived from CLF or XLF laser shots. FD data used for cosmic 
ray publications is required to be collected during hours with T aer (3km a.g.l.)< 0.1. 
FD data recorded during hours without T aer (h) profiles due to extremely poor viewing 
conditions or technical problems are not used for publications. This database spans 
nearly 10 years. 

The molecular database contains the atmospheric state variables measured on the 
ground by the weather stations and the vertical profiles derived from BLS weather bal¬ 
loons and more recently from the Global Data Assimilation System, provided from 
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NOAA’s 5 National Centers for Environmental Prediction (NCEP) [77], These atmo¬ 
spheric data are available for a position close to the Auger array in three hour intervals. 
Data updates can be obtained once per week and are filled automatically into the Auger 
atmospheric monitoring database. Such atmospheric state variables are applied as stan¬ 
dard in the air shower reconstruction, mainly during the description of the fluorescence 
light emission and the Rayleigh scattering of that light on its path between emission 
point at the air shower and the fluorescence telescopes, since mid 2011. 

9.6. Interdisciplinary atmospheric measurements 

Through its secondary role as an Earth observatory, the interdisciplinary science 
program of the Observatory is quite extensive [98]. For example, the FD has turned 
out to be the world’s best detector for measuring atmospheric transient luminous events 
known as ELVES (see Section 14.2) [99], 

To characterize local aerosol particles, an Andersen-Graseby 240 dichotomous 
sampler was installed at the Coihueco FD building for 6 months. The particles col¬ 
lected were studied to determine their sizes and shapes. Low aerosol concentrations 
were observed during the winter with an increasing concentration in spring. An ele¬ 
mental composition analysis was also performed [100]. 

The HYhrid Single Particle Lagrangian Integrated Trajectory HySPLIT model, 
(developed from NOAA), that estimates the trajectories of air mass displacements was 
used to study the source of aerosols at the Observatory. A possible correlation was ob¬ 
served between clear conditions at the Observatory, determined from measured T aer (/i) 
profiles, and air mass sources at the Pacific ocean [101]. 

10. Hybrid event reconstruction 

The hybrid reconstruction is based on fluorescence detector data with additional 
timing information from the surface detector. In the following, the individual recon¬ 
struction steps will be described. 

10.1. Pulse reconstruction at the FD 

At the beginning of the reconstruction, the baseline is subtracted from the ADC 
trace of each pixel and the background noise is estimated from the variance of the 
ADC signals at early time bins that are free from any shower signal. Each ADC count 
is then converted to photons at the aperture using the calibration constants obtained 
from the drum and relative calibrations. 

Subsequently, each triggered FD pixel is searched for a shower signal by scanning 
the signal trace for pulse start and stop times that maximize the signal to noise ratio. 
Only pulses with a signal to noise ratio > 5 are considered in the geometrical recon¬ 
struction. 


s National Oceanic and Atmospheric Administration. 
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The pulse time of the z'th pixel is given by the centroid time (“signal weighted time”) 
of all trace bins belonging to the pulse 


ti — 



( 2 ) 


where the sum runs over the time bins defined by the aforementioned signal to noise 
maximization, t* and are the time and charge of the kth ADC bin, respectively, and 
the pixels’ integrated signal is given by 


9 « = L s t- 


(3) 


The uncertainties of q\ and r, are obtained by propagating the noise variance and Pois- 
sonian photoelectron fluctuations into equations (2) and (3). 


10.2. Shower detector plane 

The shower detector plane is the plane containing the shower axis and the trig¬ 
gered fluorescence telescope. It can be reconstructed from the data of a telescope by 
minimizing 


S =J 

Lqii 


% — arccos (pi ■ h^ dp ) 
CSDP 


(4) 


over all pulses i, with the two free parameters 0sdp and 0sdp to define the vector /z SD[> 
normal to the plane in spherical coordinates and the pixel pointing direction p,. The 
pointing uncertainty for the SDP fit, <7sdp, was determined to be 0.35° by studying SDP 
fits of CLF laser shots with a known geometry. The normalization of the fit function S 
to (7 sdp allows one to interpret it as a "/} function and to derive the uncertainties of the 
SDP parameters from the 5+1 contours. 


10.3. Hybrid time fit 

From the perspective of a telescope, the projection of a shower on the camera 
evolves along the SDP. Each pulse pixel can be associated with an angle Xi along the 
SDP with respect to the horizontal axis at the telescope (see Figure 32). The angular 
movement of the shower within the SDP in this representation is [102] 

fte) = r 0 + ^tan(^). (5) 

To determine the three free parameters 7o, K p and Xu the minimum of the function 

„,2 V- (ti-t(Xi)) 2 , {tSD-t(XSD)) 2 
x ^ (6) 
has to be found [34, 35], The sum runs over all pulse pixels i with the centroid pulse 
time tj and the associated uncertainty cr(f,), adding the additional SD station time fsD 
with the uncertainty cr(fsD)- The shower front containing the surface detector station 
meets the (trial) shower axis at a point that would be seen at angle XsD, and tfxsD) is 
the expected time when the shower center would pass that point. 

An example of an event that has been observed by four telescopes is shown in Fig¬ 
ure 33. The individual four reconstructions of the geometry using the hybrid approach 
are indicated by black lines. 
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Figure 32: Geometry of an air shower within the shower detector plane. 


10.4. Light collection 

The total light at the aperture as a function of time is obtained by adding the signals 
s of the camera pixels j at each time bin i. For this purpose, only the pixels with 
pointing directions that are within a certain angular distance £ to the vector from the 
telescope to the shower center at time i are included. £ is chosen such that the signal 
to noise ratio is maximized. The light flux arriving at the detector in time bin i is 


Ft 


1 

'Alia 


^pix 

E A/> 

j= 1 


(7) 


where the sum runs over all pixels N p - ]x within £ at time bin i and is the area of the 
diaphragm opening. 


10.5. Profile reconstruction 

Once the geometry of the shower is known, the light collected at the aperture as a 
function of time can be converted to the energy deposited by the shower as a function 
of slant depth. For this purpose, the light attenuation from the shower to the detector 
needs to be accounted for and all contributing light sources need to be disentangled: 
fluorescence light [81], direct and scattered Cherenkov light [103, 104] as well as mul¬ 
tiply scattered light [105-107]. 

The proportionality between the fluorescence intensity and the energy deposit is 
given by the fluorescence yield. A good knowledge of its absolute value as well as its 
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Figure 33: Geometry reconstruction of an event observed by four telescopes and the surface detector. 


dependence on wavelength, temperature, pressure and humidity is essential to recon¬ 
struct the longitudinal profile. We use the most precise of the measurements available 
to date (cf. [108]) as provided by the Airfly Collaboration [109, 1 10]. 

The Cherenkov and fluorescence light produced by an air shower are connected to 
the energy deposit by a linear set of equations and therefore the shower profile is ob¬ 
tained by an analytic linear least squares minimization [111]. Due to the lateral extent 
of air showers, a small fraction of shower light is not contained within the optimal light 
collection area. To correct this, the universal lateral fluorescence [112] and Cherenkov 
light [113] distributions must be taken into account. The full longitudinal energy de¬ 
posit profile and its maximum (gf ) max at depth X = A max are estimated by fitting a 
Gaisser-Hillas function [114], 



^max —X 

e a 


( 8 ) 


to the photoelectrons detected in the PMTs of the FD cameras. For this purpose, a log- 
likelihood fit is used in which the number of photoelectrons detected by the PMTs of 
the FD cameras is compared to the expectation from equation (8) after folding it with 
the light yields, atmospheric transmission, lateral distributions and detector response. 
The two shape parameters Xo and A are constrained to their average values to allow for 
a gradual transition from a two- to a four-parameter fit depending on the observed track 
length and number of detected photons of the respective event (cf. [1 1 1]). 
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(a) Light at aperture. 



slant depth [g/cm ] 


(b) Energy deposit. 


Figure 34: Example of a reconstructed shower profile. 


Finally, the calorimetric energy of the shower is obtained by integrating equa¬ 
tion (8) and the total energy is estimated by correcting for the ‘invisible energy’ carried 
away by neutrinos and high energy muons [115]. An example of the measured light at 
aperture and the reconstructed light contributions, and energy deposit profile is shown 
in Figs. 34(a) and 34(b). 

11. SD event reconstruction 

The reconstruction of the energy and the arrival direction of the cosmic rays pro¬ 
ducing air showers that have triggered the surface detector array is based on the sizes 
and times of signals registered from individual SD stations. At the highest energies, 
above lOEeV, the footprint of the air shower on the ground extends over more than 
25 km 2 . By sampling both the arrival times and the deposited signal in the detector 
array, the shower geometry, i.e., the shower core, the arrival direction of the incident 
cosmic ray, and the shower size can be determined. 

11.1. Event selection 

To ensure good data quality for physics analysis there are two additional off-line 
triggers. The physics trigger, T4, is needed to select real showers from the set of 
stored T3 data (see Section 6.3) that also contain background signals from low energy 
air showers. This trigger is mainly based on a coincidence between adjacent detector 
stations within the propagation time of the shower front. In selected events, random 
stations are identified by their time incompatibility with the estimated shower front. 
The time cuts were determined such that 99 % of the stations containing a physical 
signal from the shower are kept. An algorithm for the signal search in the time traces 
is used to reject signals produced by random muons by searching for time-compatible 
peaks. 

To guarantee the selection of well-contained events, a fiducial cut (called the 6T5 
trigger) is applied so that only events in which the station with the highest signal is 
surrounded by all 6 operating neighbors (i.e., a working hexagon) are accepted. This 
condition assures an accurate reconstruction of the impact point on the ground, and at 
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Figure 35: Reconstruction of shower geometry: schematic representation of the evolution of the shower 
front. 



Figure 36: Reconstruction of shower geometry: dependence of signal start times (relative to the timing of a 
plane shower front) on perpendicular distance to the shower axis. The shaded line is the resulting fit of the 
evolution model and its uncertainty. 


the same time allowing for a simple geometrical calculation of the aperture/exposure 
[36], important for, e.g., the spectrum analysis [38]. For arrival-direction studies a less 
strict cut can be used (5T5 or even 4T5). 

11.2. Shower geometry 

A rough approximation for the arrival direction of the shower is obtained by fitting 
the start times of the signals, f ; , in individual SD stations to a plane front. For events 
with enough triggered stations, these times are described by a more detailed concentric- 
spherical model, see Figure 35, which approximates the evolution of the shower front 
with a speed-of-light inflating sphere, 

c(ti~t 0 ) = |x s h-x,-|, (9) 

where x, are positions of the stations on the ground and where x s h and to are a vir¬ 
tual origin and a start-time of the shower development (see Figure 36). From this 
4-parameter fit the radius of curvature of the inflating sphere is determined from the 
time at which the core of the shower is inferred to hit the ground. 

11.3. Lateral distribution function 

The impact points of the air showers on the ground, x^, are obtained from fits of 
the signals in SD stations. This fit of the lateral distribution function (LDF) is based 
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Figure 37: Example of signal sizes an extensive air shower induces in the stations of the surface detector 
array. Note that the spacing of the regular grid is 1.5 km. Colors represent the arrival time of the shower 
front from early (yellow) to late (red) and the size of the markers is proportional to the logarithm of the 
signal. The line represents the shower arrival direction. 


on a maximum likelihood method which also takes into account the probabilities for 
the stations that did not trigger and the stations close to the shower axis with saturated 
signal traces. The saturation is caused by the overflow of the FADC read-out electronics 
with finite dynamic range and a modification of the signal due to the transition of the 
PMTs from a linear to a non-linear behavior. In the majority of cases the missing part 
of the signals are recovered using the procedure described in [116]. 

An example of the footprint on the array of an event produced by a cosmic ray with 
an energy of (104 ± 11) EeV and a zenith angle of (25.1 ±0.1) degrees is shown in 
Figure 37. The lateral distribution of the signals is depicted in Figure 38. The function 
employed to describe the lateral distribution of the signals on the ground is a modified 
Nishimura-Kamata-Greisen function [117, 118], 

where r opt is the optimum distance, r\ = 700 m and 5(r op t) is an estimator of the shower 
size used in an energy assignment. For the SD array with station spacing of 1.5 km the 
optimum distance [119] is r opt = 1000 m and the shower size is thus 5(1000). The pa¬ 
rameter p depends on the zenith angle and shower size. Events up to zenith angle 60° 
are observed at an earlier shower age than more inclined ones, thus having a steeper 
LDF due to the different contributions from the muonic and the electromagnetic com¬ 
ponents at the ground. For events with only 3 stations, the reconstruction of the air 
showers can be obtained only by fixing the two parameters, p and y to a parametriza- 
tion obtained using events with a number of stations larger than 4. 

The reconstruction accuracy of 5(1000), G s; 10 oo)> is composed of three contribu¬ 
tions: a statistical uncertainty due to the finite number of particles producing a signal in 
a given SD station and the limited dynamic range of the signal detection; a systematic 
uncertainty due to assumptions on the shape of the lateral distribution function; and 
an uncertainty due to shower-to-shower fluctuations [120], The last term contributes a 
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Figure 38: Dependence of the signal size on distance from the shower core. 



Figure 39: Angular resolution as a function of the zenith angle 6 for events with an energy above 3 EeV, and 
for various station multiplicities. [40]. 


factor of about 10%, while the contribution of the first two terms depends on energy 
and varies from 20 % (at low energies) to 6 % (at the highest energies). 


11.4. Shower arrival direction 

Shower axis a is obtained from the virtual shower origin (of the geometrical recon¬ 
struction) and the shower impact point on the ground (from the LDF reconstruction). 


a = 


■Ah ^gr 
l^sh Vgr 


(ID 


To estimate an angular resolution of the whole reconstruction procedure a single 
station time variance is modeled [121] to take into account the size of the total signal 
and the time evolution of the signal trace. As shown in Figure 39, the angular resolution 
achieved for events with more than three stations is better than 1.6°, and better than 0.9° 
for events with more than six stations [40]. 


11.5. Energy calibration 

For a given energy, the value of 5(1000) decreases with the zenith angle 6 due to the 
attenuation of the shower particles and geometrical effects. Assuming an isotropic flux 
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Figure 40: Attenuation curve described by a third degree polynomial in x = cos 2 0 — cos 2 0 where 0 = 38° 
(denoted by the dashed vertical line). In this example the polynomial coefficients are deduced from 5(1000) 
dependence at 538 ~ 50 VEM which corresponds to an energy of about 10.5 EeV. 


of primary cosmic rays at the top of the atmosphere, we extract the shape of the attenua¬ 
tion curve (see Figure 40) from the data using the Constant Intensity Cut (CIC) method 
[39], The attenuation curve /cic(®) has been fitted with a third degree polynomial 
in x = cos 2 0 — cos 2 0, i.e., fcic(0) = 1 + ax + bx 2 +cx 3 , where a = 0.980 ± 0.004, 
b = -1.68 ±0.01, and c = -1.30±0.45 [11], 

The median angle, 0 = 38°, is taken as a reference point to convert S( 1000) to 
$38 = 5(lOOO)//cic(0)- ^38 may be regarded as the signal a particular shower with 
size 5(1000) would have produced had it arrived at 0 = 38°. 

To estimate the energy of the primary particle producing the air-showers recorded 
with the SD, the advantage comes from the hybrid detection: the air-showers that have 
triggered independently the FD and SD are used for the cross-calibration. High-quality 
hybrid events, as defined below, with reconstructed zenith angles less than 60° are used 
to relate the shower size from SD to the almost-calorimetric measurement of the shower 
energy from FD, Epp>. 

These hybrid events must be such that the reconstruction of an energy estimator 
can be derived independently from both the SD and FD parts of the event [1 15, 122], 

Only a subsample of events that passes strict quality and field of view cuts is used. 
For the FD part of the event, we require an accurate fit of the longitudinal profile to the 
Gaisser-Hillas function. Furthermore, the depth of the shower maximum, 2f max , must 
be contained within the telescope field-of-view and measured with an accuracy better 
than 40g/cm 2 . 

The uncertainty on the reconstructed Epp, is required to be less than 18 %. The final 
criteria for defining the calibration data sample include a selection of clear atmosphere 
conditions based on the measurements of the vertical aerosol optical depth, and on the 
cloud fraction measured by the lidar systems of the Observatory. To avoid any potential 
bias of the event selection on the mass of the primary particle, a fiducial cut on the slant 
depth range observed by the telescopes is also added, ensuring that the field of view is 
large enough to observe all plausible values of X max for the geometry of each individual 
shower [122], 

The final step in the calibration analysis leads to a relation between .S';8 and Epp,. 
The 1475 high quality hybrid events recorded between Jan 2004 and Dec 2012 which 
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Figure 41: Correlation between S 38 and Efd [11, 122]. 


have an energy above the SD full efficiency trigger threshold [36] are used in the cali¬ 
bration. The correlation between the two variables is obtained from a maximum like¬ 
lihood method [122, 123] which takes into account the evolution of uncertainties with 
energy, as well as event migrations due to the finite energy resolution of the SD. The 
relation between and Efd is well described by a single power-law function, 

£fd=A(S 38 /VEM) b (12) 

where the resulting parameters from the data fit are A = (1.90 ±0.05) x 10 17 eV and 
B = 1.025 ±0.007 [11, 124], As can be seen in Figure 41, the most energetic event 
used in this analysis has an energy of 79 EeV. 

The resolution of the final SD energy estimator, 

£sd = A(5(1OOO)// C ic( 0)/VEM) b , (13) 

can be inferred from the distribution of the ratio T’sd/Efd- Using the FD energy res¬ 
olution of 7.6 %, the resulting SD energy resolution with its statistical uncertainty is 
Ge sd /Esd = (16±l)%at the lower energy edge in Figure 41 and (12± 1)% at the high¬ 
est energies. Due to the large number of events accumulated until December 2012, the 
systematic uncertainty on the SD energy due to the calibration is better than 2 % over 
the whole energy range. The systematic uncertainties in the energy scale, shown in Ta¬ 
ble 3, are dominated by the absolute FD calibration [124]. Further consistency checks 
are performed by joint calibration campaigns with the Telescope Array [125, 126]. 

The dataset recorded extends up to larger angles of 90°. For the inclined events, 
with zenith angles larger than 60° we employ a different reconstruction method. More 
details on the reconstruction of inclined events can be found in [123, 127, 128]. The 
energy range of full efficiency of the surface detector has been extended down to 
3xl0 17 eV using the events recorded by the 750 m array (see Section 13). The re¬ 
construction of this subsample of events is described in [128-130]. 


65 



Table 3: Systematic uncertainties in the energy scale. 


Absolute fluorescence yield 

3.4% 

Fluorescence spectrum and quenching parameters 

1 .1% 

Subtotal, Fluorescence yield 

3.6% 

Aerosol optical depth 

3-6% 

Aerosol phase function 

1 % 

Wavelength dependence of aerosol scattering 

0.5% 

Atmospheric density profile 

1 % 

Subtotal, Atmosphere 

3.4-6.2 % 

Absolute FD calibration 

9% 

Nightly relative calibration 

2 % 

Optical efficiency 

3.5% 

Subtotal, FD calibration 

9.9% 

Folding with point spread function 

5% 

Multiple scattering model 

1 % 

Simulation bias 

2 % 

Constraints in the Gaisser-Hillas fit 

3.5-1 % 

Subtotal, FD profile reconstruction 

6.5-5.6 % 

Invisible energy 

3-1.5 % 

Statistical error of SD calibration fit 

0.7-1.8 % 

Stability of the energy scale 

5% 

Total 

14% 


12. Performance characteristics of the Observatory 

12.1. Key performance parameters 

In Table 4 are summarized some of the important parameters that characterize the 
performance of the Observatory. These parameters include the event rate of the detec¬ 
tors and the resolutions of the different reconstructed observables. 

12.2. Surface detector performance 

Stable data taking with the surface detector array started in lanuary 2004 and the 
Observatory has been running in its full configuration since 2008. As described in 
Section 7, various parameters are continuously monitored to optimize the performance 
of the detectors and ensure reliable data. 

The monitoring tool includes so-called performance metrics to monitor the overall 
performance of the surface detector array. Relevant data useful for long term studies 
and for quality checks are stored in the Auger Monitoring database on a one-day basis. 
For example, mean values over one day of the number of active SD detectors and 


66 












Table 4: Key performance parameters for the Auger Observatory. 


SD 

SD Annual Exposure 

~5500 km 2 sryr 

T3 rate 

0.1 Hz 

T5 events/yr, E > 3 EeV 

~ 14,500 

T5 events/yr, E > 10 EeV 

~1500 

Reconstruction accuracy (Siooo) 22 % (low E) to 12 % (high E) 

Angular resolution 

1.6° (3 stations) 


0.9° (>5 stations) 

Energy resolution 

16 % (low E) to 12 % (high E) 

FD 

On-time 

~ 15% 

Rate per building 

0.012 Hz 

Rate per HEAT 

0.026 Hz 

Hybrid 

Core resolution 

50 m 

Angular resolution 

0 .6° 

Energy resolution (FD) 

8 % 

Amax resolution 

<20 g/cm 2 


the number of active hexagons as well as the nominal value (expected value if all the 
detectors deployed were active) are available. As an example. Figure 42 shows the 
number of active SD stations normalized to the nominal number of stations in the array 
for the last 4 years. This plot is a convolution of the status of the active stations and of 
the efficiency of the CDAS, which since the beginning is better than 99.5 %. 

Figure 43 shows the number of active hexagons for the same period. This variable 
is a key parameter since it is the basis of the exposure evaluation. Indeed, the offline T5 
fiducial trigger, described in Section 11 selects only events for which the hottest station 
is surrounded by an active hexagon. Thus, above 3 x 10 18 eV, when the full efficiency 
of detection of the array is reached (at least three triggered tanks), the exposure is 
simply proportional to the integrated number of active hexagons during the period. The 
number of active hexagons fluctuates because of intermittent outages in electronics, 
communications, weather, and other factors [36]. Larger numbers of hexagons can be 
affected when the problems occur at the WLAN sector level. These outages can usually 
be resolved quickly. 

The rate of events (T5 events) normalized to the average number of active hexagons 
is expected to be stable in time above the energy threshold of 3 x 10 18 eV, which can 
be seen in Figure 44. 

Finally the integrated exposure between 1 January 2004 and 31 December 2012 is 
shown in Figure 45. Since completion of the array in 2008, the increase of the exposure 
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Figure 42: Number of active SD stations normalized to the nominal number of SD stations in the array, as a 
function of time. (Note: WCD = water Cherenkov detector.) 



Time 


Figure 43: Number of active hexagons as a function of time. 



Figure 44: Evolution of the daily T5 rate normalized to the number of hexagons for the period 2008-2012. 
The data gap in 2009 corresponds to a period during which the communications system experienced technical 
problems. 


has been about 5500 km 2 sr per year. 
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Figure 45: Evolution of the exposure between 1 January 2004 and 31 December 2012. The lack of exposure 
growth during an interval in 2009 corresponds to the data gap seen in Figure 44. 


12.3. Fluorescence detector performance 

The data taking of the FD can only take place under specific environmental condi¬ 
tions and is organized in night shifts. As described in Section 4.5, the telescopes are 
not operated when the weather conditions are unfavorable (high wind speed, rain, snow, 
etc.) and when the observed sky brightness (caused mainly by scattered moonlight) is 
too high. As a consequence, the shifters have to continuously monitor (see Section 7.4) 
the atmospheric and environmental conditions and judge the operation mode on the 
basis of the available information. 

The performance of the fluorescence and hybrid data taking is then influenced by 
many effects. These can be external, e.g., lightning or storms, or internal to the data 
taking itself, e.g., DAQ failures. For the determination of the on-time of the Obser¬ 
vatory in the hybrid detection mode it is, therefore, crucial to take into account all of 
these occurrences and derive a solid description of the data taking time sequence. 

Data losses and inefficiencies can occur on different levels, from the smallest unit 
of the FD, i.e., one single photomultiplier (pixel) readout channel, up to the highest 
level, i.e., the combined SD/FD data taking of the Observatory. 

The active time of the FD data acquisition is calculated using a minimum bias data 
stream with a less restrictive trigger condition. Since July 2007, the relevant informa¬ 
tion concerning the status of the FD detector has been read out from the Observatory 
monitoring system (see Section 7.4). An on-time dedicated database has been set up by 
storing the average variances and the on-time fraction of individual telescopes in time 
bins of 10 minutes. The information on the veto due to the operation of the lidar or to 
an anomalous trigger rate on FD together with the status of the CDAS needed to form 
a hybrid event are also recorded. The method to calculate the on-time of the hybrid 
detector is described in detail in Ref. [37]. 

The accumulated on-time is shown in Figure 46, top, for the six telescopes at 
Coihueco and for the three HEAT telescopes. The average FD on-time (full circles) 
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Figure 46: Top: accumulated on-time since 1 Jul 2007 for the 6 telescopes at Coihueco and for the 3 HEAT 
telescopes. Bottom: FD and hybrid on-time of individual telescopes since 1 Jan 2011. (1-6), (7-12), (13- 
18), (19-24), (25-27) for the sites of Los Leones, Los Morados, Loma Amarilla, Coihueco and HEAT, 
respectively. 


of individual telescopes since 1 January 2011 is shown in Figure 46, bottom. Requir¬ 
ing that the CDAS is active defines the hybrid on-time (empty circles). 

The time evolution of the full hybrid duty cycle over 9 years of operation is shown 
in Figure 47, top, for all FD sites. Time bins are taken as the time intervals elapsed 
between two subsequent FD data taking shifts. The performance of the hybrid detector 
is compared to the nominal DAQ time (see Section 4.5) in the top panel of Figure 47. 
In the bottom panel, the FD on-time is normalized to the time with high voltage ON, 
leading to an average FD detector readiness of about 85 % for all telescopes. The re¬ 
maining inefficiency can be ascribed to different factors such as bad weather conditions 
(high wind load and/or rain) or high variances due to bright stars/planets crossing the 
field of view of the FD. 

It should be noted that the FD site of Los Morados became operational in May 
2005, Loma Amarilla starting from March 2007 and HEAT since September 2009. 
After the initial phase due to the start up of the running operations, the mean on-time is 
about 15 % for all of the FD sites. Additionally, a seasonal modulation is visible, since 
higher on-time fractions are observed in the austral winter during which the nights are 
longer. 
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Figure 47: Top: time evolution of the average hybrid on-time fraction over nine years of operation of the 
Observatory. The thick gray line defines the scheduled data taking time fraction defined as the time periods 
with moon fraction lower than 70 % and with the moon being below the horizon for more than 3 hours (see 
also Section 4.5 for details). Bottom: readiness of the FD detector (see text for details). 



Figure 48: Daily rate of hybrid reconstructed events as a function of year, starting in 2005, for (from top to 
bottom) Los Leones, Los Morados, Loma Amarilla and Coihueco, respectively. 


12.4. Time stability of the hybrid detector response 

The performance of the hybrid detector is demonstrated as a function of time us¬ 
ing a sample of events fulfilling basic reconstruction requirements, such as a reliable 
geometrical reconstruction and accurate longitudinal profile and energy measurement. 
The daily rate of well-reconstructed hybrid events observed by individual FD sites is 
shown in Figure 48 as a function of time, starting in 2005. 

An important benchmark for the time stability of the hybrid detector response is 
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Figure 49: Top: fraction of all events that are well reconstructed hybrids since 2010. Bottom: mean energy 
for reconstructed hybrid events. 


the study of the effective on-time, defined as the fraction of all events that are well 
reconstructed hybrids. Its time evolution, shown in Figure 49 (top), exhibits quite 
a stable behavior over time. Moreover the mean energy of the hybrid events above 
10 18 eV, with distance to the shower maximum between 7 and 25 km (corresponding to 
the 90 % of the entire hybrid data sample), is shown as a function of time in Figure 49 
(bottom). All these features demonstrate the quality of the collected hybrid data and 
directly assess their long term stability. 

13. Enhancements to the Auger Observatory 

With the simultaneous and successful operation of the SD and FD, the Pierre Auger 
Collaboration has demonstrated the power of hybrid measurements. Since completion 
of the baseline construction, new enhancements were proposed to further extend the 
science reach of the Observatory. 

13.1. High Elevation Auger Telescopes (HEAT) 

Three additional fluorescence telescopes with an elevated field of view were built 
about 180m in front of the FD site at Coihueco [131]. These telescopes are very 
similar to the original fluorescence telescopes but can be tilted by 29° upward with 
an electrically driven hydraulic system. These three telescopes work independently of 
other FD sites and form the “fifth site” of the Observatory. The HEAT telescopes were 
designed to cover the elevation range from 30° to 58°, which lies above the field of 
view of the other FD telescopes. The HEAT telescopes allow a determination of the 
cosmic ray spectrum and X max distributions in the energy range from below the second 
knee up to the ankle. The HEAT telescopes are depicted in Figures 50 and 51. 
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Figure 50: Photo of the three HEAT telescopes in tilted mode. The container for DAQ, slow control, and 
calibration hardware is behind the enclosure of the second telescope. 



(a) Horizontal (downward) mode for service and (b) Data taking (upward) mode in tilted orientation, 
cross-calibration. 


Figure 51: Schematic view of the cross-section of one of the HEAT telescopes. 


The main objective of this extension was to lower the energy threshold of hybrid 
data to enable an unbiased detection of nearby low-energy showers. In combination 
with the SD, information from an infilled array of water Cherenkov detectors on a 
750 m grid (see Section 13.2) close to the HEAT site, the energy range of high quality 
hybrid air shower measurements has been extended down to 10 17 eV. In addition, close 
inclined shower events are detectable without any mass dependent bias or cuts up to 
the highest energies. 

The layout of HEAT consists of three telescope enclosures and one container for 
DAQ, slow control, and calibration hardware (see Figure 50). The telescope enclosure 
consists of three main building blocks. First is the concrete foundation that supports the 
tilting mechanism and provides stability for the whole building. Second is the strong 
steel base plate, filled with concrete, on which all the sensitive optical elements are 
mounted. Finally a relatively lightweight steel container encloses the optical compo¬ 
nents and electronics. The base plate is connected to the foundation, while the steel 
enclosure is itself fixed to the plate. Similarly to the baseline telescopes, a shutter 
system is mounted on the steel enclosure, but of a different design. 

The HEAT telescopes can be tilted using the hydraulic mechanism. The telescopes 
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are parked in the horizontal position between the FD data taking periods to be ac¬ 
cessible for maintenance, see Figure 51. The same position is used for the absolute 
calibration of the HEAT telescopes and also for the cross-calibration with telescopes at 
Coihueco. All three HEAT telescopes are usually moved in the upward position before 
the first DAQ night and stay there during the whole data taking period. 

To ensure sufficient mechanical stability during high winds and snow loads, all 
telescope components are connected to a heavy and stiff ground plate with adjustment 
mounting bolts. The mechanical stability is monitored by two types of sensors. The 
first type is the inclination sensor that is used to measure the inclination variations at 
different points inside the HEAT shelters. The second type is needed to measure the 
distance variations between the optical components of the telescope. The distance sen¬ 
sors are not only used to measure the long term variations, but also the higher frequency 
variations that can take place in the telescope when subjected to strong winds or other 
similar conditions. The maximal allowed deformations and any movements after tilting 
lead to an angular offset less than 0.1°. 

The response of the HEAT cameras was tested at multiple elevations using the 
relative calibration method (see Section 4.4.3). The effect on the signal of tilting HEAT 
is at the percent level or below, which matches the overall magnitude expected due to 
the direction of the Earth’s magnetic field as seen by the PMTs. Also, the absolute 
calibration may be determined in the horizontal mode. 

The Schmidt optics of the HEAT telescopes, camera body, PMTs, light collectors, 
etc., are the same as in the other sites. All three spherical mirrors are built up from 
hexagonal glass mirrors with vacuum-deposited reflective coatings. 

A feature that sets HEAT apart from the classic Auger telescopes is its new elec¬ 
tronics kit that can sample up to 40 MHz instead of 10 MHz. In practice, a sampling 
rate of 20 MHz (corresponding to a 50 ns FADC bin size) was chosen. The higher rate 
improves the measurement for close showers that have a correspondingly larger angu¬ 
lar velocity - precisely the showers we are interested in observing with HEAT. From 
this it follows that the first level trigger interval was reduced to 50 ns, whereas the sec¬ 
ond level trigger continues to operate every 100 ns. The length (in time) of the FADC 
traces remains the same, so the number of bins doubles. 

The trigger rate of the HEAT telescopes is high, particularly because of the Cherenkov 
light from low energy showers. Therefore the T4 trigger has been implemented to re¬ 
duce the readout of the SD array for these low energy showers. 

13.2. Auger Muon and Infilled Ground Array (AMIGA) 

A dedicated detector to directly measure the muon content of air showers is being 
built. The AMIGA enhancement [132-134] is a joint system of water Cherenkov and 
buried scintillator detectors that spans an area of 23.5 km 2 in a denser array with 750 m 
spacing nested within the 1500 m array (see Figure 52). The area is centered 6 km 
away from the Coihueco fluorescence site. The 750 m array is fully efficient from 
3 x 10 17 eV onwards for air showers with zenith angle < 55° [129]. Although the 
infilled area is much smaller than the regular SD, the flux of cosmic rays increases 
steeply with decreasing energy such that this area is sufficient to observe a significant 
number of events and to study the region between the second knee [135] and the ankle 
of the cosmic ray spectrum. 
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Figure 52: AMIGA layout: an infill of surface stations with an inter-detector spacing of 750 m plus plastic 
scintillators of 30 m 2 buried under « 540g/cm 2 of vertical mass to measure the muon component of the 
showers. The small shaded area indicates the prototype hexagon (Unitary Cell) of the muon detector which 
has been fully operational since March 2015. Two positions of UC are equipped with extra twin scintillators 
to allow the detector accuracy to be assessed while in the hexagon’s centre also 20 m 2 were installed at 
~ 310g/cm 2 to experimentally analyze the shielding power of the local soil. 



Figure 53: AMIGA station: SD+MD paired detectors. During the MD prototype phase the 30m 2 buried 
scintillators are segmented into 4 modules, 2 x 10m 2 + 2 x5m’. To avoid shadowing effects by the water 
Cherenkov detector, there are 5 m of sideways clearance. The buried front end electronics is serviceable by 
means of an access pipe which is filled with local soil bags. Data are sent by a dedicated WiFi antenna. 


The SD 750 m array was completed in September 2011 while the first prototype 
hexagon of buried scintillators, the Unitary Cell, has been fully operational since March 
2015. This engineering array consists of seven water Cherenkov detectors paired with 
30m 2 scintillators segmented in two modules of 10m 2 plus two of 5 m 2 in each po¬ 
sition. In addition, two positions of the hexagon were equipped with twin detectors 
(extra 30m 2 scintillators) to allow the accuracy of the muon counting technique to be 
experimentally assessed [136] and one position has 20m 2 of extra scintillators buried 
at a shallower depth to analyze the shielding features. In total, 290m 2 fully equipped 
plastic scintillators are operative in the Unitary Cell. The proven tools and methods 
used for the analysis of the 1500 m SD array data have been extended to reconstruct 
the lower energy events. The angular resolution for E> 4 x 10 17 eV is better than 1° 
and the energy reconstruction is based on the lateral density of shower particles at the 
optimal distance of 450 m from the core [130]. 

The buried scintillators are the core of the detection system for the muonic compo¬ 
nent of air showers (the muon detector, MD). To effectively shield the electromagnetic 
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Figure 54: AMIGA scintillator detector, illustrating the assembly of a 10 m 2 module. Strips are grouped 
in two sets of 32 strips on each side of the electronics dome located at the center of the detector. The inset 
shows a detailed view of the manifold fiber routing and optical connector. The multi-anode PMT and front 
end electronics board are hosted in the central dome. Once deployed, access to the buried setup is provided 
by an inspection pipe. 


component, the MD is placed under sa 540g/cm 2 of vertical mass corresponding to a 
depth of 2.3 m in the local soil (sa 20 radiation lengths) while the shallower extra scintil¬ 
lators are at « 310g/cm 2 (1.3 m). These shieldings impose a cutoff for vertical muons 
of around 1 GeV and 0.6 GeV respectively. The layout of SD+MD paired stations is 
shown in Figure 53. The scintillator surface of each MD station is highly segmented. 
It consists of modules made of 64 strips each. Strips are 4.1cm widex 1.0cm thick 
and 400 cm or 200 cm long for the 10 m 2 and 5 m 2 modules, respectively. They con¬ 
sist of extruded Dow Styron 663W polystyrene doped by weight with 1% PPO (2,5- 
diphenyloxazole) and 0.03% POPOP (l,4-bis(5-phenyloxazole-2-yl)benzene). They 
are completely wrapped with a thin white reflective layer of titanium dioxide (TiOo) 
except for a central groove into which a wavelength shifting (WLS) optical fiber is 
installed. The light output uniformity is ±5%. Because the scintillators have an at¬ 
tenuation length of ~(55 ±5) mm, light is transported to a photomultiplier tube using 
the WLS fiber. The manifold of fibers of each module ends in an optical connector 
matched to a 64 multi-anode PMT from the Hamamatsu H8804 series. Scintillator 
strips are grouped in two sets of 32 strips on each side of the PMT and front end elec¬ 
tronics board (see Figure 54). 

The bandwidth of the front end electronics is set to 180 MHz to determine the pulse 
width. Signal sampling is performed by a Field Programmable Gate Array (FPGA) 
from the ALTERA Cyclone III series at 320 MHz. MD scintillator modules receive 
the trigger signal from their associated SD station. The lowest level trigger (Tl) of 
the surface detectors is used. Once a Tl condition is fulfilled on the surface, its MD 
companion freezes a 6.4 |_is data sample into a local buffer - 1.6 p.s before and 4.8 p.s 
after the Tl. Data are then moved to an external RAM capable of storing 1024 triggers 
[137]. 

Incoming analog signals from each pixel of the PMT are digitized with a discrim¬ 
inator that provides the input to the FPGA. Samples can be either a logical “1” or “0” 
depending on whether the incoming signal was above or below a given (programmable) 
discrimination threshold. This method of one-bit resolution is very robust for counting 
muons in a highly segmented detector. This avoids missing muons due to simultaneous 
particle arrivals [138], It relies neither on deconvolving the number of muons from an 
integrated signal, nor on the PMT gain or its fluctuations, nor on the point of impact 
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of the muon and the corresponding light attenuation along the fiber. It also does not 
require a thick scintillator to control Poissonian fluctuations in the number of single 
photoelectron pulses per impinging muon [139]. The MD station power is supplied by 
an additional solar panel and battery box (see Figure 53). 

Whereas the data of the 750 m array are transmitted over the same SD radio as 
for the regular array, a dedicated telecommunication system based on WiFi 802. llg 
standard is used for MD data transmission during the prototype phase. The system is 
provided by an extra antenna located on the SD mast as indicated in figure 53. MD data 
are sent to the CDAS only at T3 level. As WiFi based telecommunication has proven to 
satisfy the network throughput and data transfer requirements for SD T2s and SD+MD 
T3s data, it is foreseen that this system will be used for the whole AMIGA detector. 

14. Other capabilities of the Observatory 

14.1. Space weather 

The rate of background low energy particles detected by the water Cherenkov de¬ 
tectors of the Observatory is recorded every second and transmitted to CDAS. This rate, 
of around 2000 particles/s per detector, is used to monitor the stability of operations. 
The particles themselves are residual components of air showers initiated by primary 
cosmic rays with a mean energy of about 90 GeV. We have observed that this rate cor¬ 
relates strongly with neutron monitors measuring Forbush decreases [140]. Measuring 
the flux of secondaries with great accuracy allows the Observatory to contribute to the 
“Space Weather” program [141]. These data are available on the Public Event Display 
of the Observatory web site [142]. 

14.2. ELVES 

ELVES (an acronym for Emissions of Light and Very low frequency perturbations 
due to Electromagnetic pulse Sources) are transient luminous events produced by heat¬ 
ing, ionization, and subsequent optical emission due to intense electromagnetic pulses 
radiated by both positive and negative lightning discharges. These intense flashes of 
light appear in the night sky as rapidly expanding quasi-circular fronts; generated at 
80-95 km altitudes, they are visible at distances of several hundred kilometers. The 
original pulse lasts less than 20 /is, but the propagating light front is visible for a few 
ms [143], The first clear observation of ELVES was made using a high-speed pho¬ 
tometer [144]; more recently such phenomena were studied using both ground based 
photometers [145-147] and satellite missions [148]. After the first serendipitous obser¬ 
vation of an ELVES events during a FD shift [99] further studies, done on a pre-scaled 
sample of data taken in the period 2008-2011, have shown that the FD is ideally suited 
for detailed studies of ELVES. A new, modified third level trigger algorithm was imple¬ 
mented in March 2013 [149] to increase the detection and recording efficiency of these 
events. Since then, a large fraction of these events is regularly recorded by two or three 
FD eyes allowing a stereo reconstruction of the light emission. Also, since January 
2014, when an ELVES trigger is received, the FD camera readout is extended to 300 
/is to allow observation of the light emitted vertically above the causative lightning. 
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Figure 55: A typical ELVES candidate event as seen in the FD. 


Auger detected 305 events in nine months of running in 2013, and 581 events (in¬ 
cluding 127 stereo and 20 triplets) in all of 2014. The distance of the causative lightning 
ranged from 300-1000 km, as determined by comparison with WWLLN data [150], 
More than 40% of these events are correlated to lightning detected by WWLLN. 

Figure 55 shows the light propagation pattern in an ELVES candidate event: the 
earliest triggered pixels are in blue, while the latest are in red. 

15. Outreach 

The scale and scope of the physics explored at the Pierre Auger Observatory offer 
significant opportunities for outreach both to the local community and beyond to the 
collaborating countries. Education, outreach and public relations have been an integral 
part of the Auger organization from the beginning when these activities were included 
as a level two management task group. The goals of the Outreach and Education Task 
are to encourage and support a wide range of efforts that link Auger scientists and 
the science of particle astrophysics, particle physics and related technologies to the 
public and especially to schools. Outreach focused on the communities surrounding 
the Observatory has fostered a remarkable amount of goodwill, which has contributed 
significantly to the success of the project. 

The Auger Collaboration initiated outreach first locally as a way to become better 
integrated into the community during the constmction phase of the Observatory. Later 
outreach activities spread to the participating institutions but on a worldwide scale, 
including the Internet. 

The heart of local outreach activities is the Auger Visitor Center (VC), located in the 
central office and data acquisition building on the Observatory campus. A staff member 
dedicated to outreach gives presentations and tours to visitors that are mostly from the 
area but often from all over Argentina and even from 25 other countries worldwide. 
Many of the visitors are in the area because of the proximity of the Las Lenas ski area 
and other area tourist attractions. Over 90,000 people have attended the lectures in the 
Visitor Center since it opened in 2001. The impact of these visits can be seen from the 
still increasing interest and the comments in the guest book. The VC, which seats up to 
50 people, is outfitted with multimedia equipment and contains a number of displays 
illustrating features of the Observatory. These displays include a full size SD station, a 
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quarter sized model of an FD mirror, a spark chamber, a Geiger counter, a number of 
posters that explain the science and detectors of the Observatory and a library of books 
in several languages. 

Many schools outside of Malargiie, some over 100 km away, have trouble bring¬ 
ing their students to the Visitor Center. A rural schools education program, funded 
by donations from the collaboration, sponsors a dedicated team of Observatory staff 
members and collaborators, who give presentations on Auger and science generally. 
The visiting team not only goes to these schools to give information on Auger but also 
helps to enhance the learning environment by providing learning materials or helping 
with infrastructure improvements such as connection to the Internet. The effect of these 
visits on the students is enormous and creates an atmosphere of good will towards the 
Observatory. 

Every two years a science fair is organized by the Observatory. The fair targets 
both elementary schools and high schools, and is still growing. The latest fair hosted 
36 entries with schools from all over Mendoza Province participating. The exhibits and 
presentations of the participants were judged by international members of the Auger 
Collaboration. The interaction of the participants with the Auger scientists reinforces 
the connection between Auger and surrounding communities. 

Each of the surface detector stations placed in the field was given a name. During 
deployment, the stations were often named after the daughter or son of the nearby 
puestero (farmer). One of the results has been that there has been a negligible amount 
of vandalism even though the detectors are spread over 3000 km 2 , with a number only 
a few meters from a road. 

Every November a collaboration meeting is scheduled to overlap with the annual 
Malargiie Days celebration and a large group of collaborators march in the celebra¬ 
tion parade behind a colorful Auger banner. These close contacts together with the 
local outreach activities have instilled a sense of ownership of the Observatory by the 
community. 

Our dedication to education is also clear from the construction of the James Cronin 
School, a secondary school in Malargiie inaugurated in 2006. The school, built from 
donated funds, was named for one of the founders of the Pierre Auger Observatory 
for his contribution to the local community. Members of the collaboration have also 
been instrumental in helping to bring a new planetarium, one of the most modern in 
Argentina, to Malargiie. 

For outreach on a larger scale the Collaboration provides cosmic ray event data 
on the public Auger Web site (www . auger . org) in nearly real time along with in¬ 
formation, photos, videos and teaching materials. This material not only explains the 
mission of the Observatory, but also contains educational material on several aspects of 
the measurement of cosmic rays and the history of these measurements. Furthermore, 
it provides manuals on how to work with and analyze the Auger public data set. The 
online event display, coupled to the public data set, is a useful tool to provide insight 
to students in what is measured and how it is interpreted. These materials are mostly 
aimed at students at the high school level or above. An online analysis interface called 
VISPA [151] has been set up to allow students to work with and analyze these data. 

Outreach has been an important part of the activities of the Auger Project. Our close 
relationship with the people of Malargiie and the other local communities as a result 
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of our outreach activities has not only made our work comfortable and rewarding but 
has, indeed, contributed to the success of the Observatory. Among the collaborating 
institutions many innovative outreach ideas have sprung from our research. Because 
we can easily show their continuous presence around us, cosmic rays are an effective 
way to excite young people about the wonders and science of the cosmos. 

16. Further developments 

Even as the Pierre Auger Observatory was under construction, new ideas for meth¬ 
ods of air shower detection were being developed. These ideas became more focused 
as the underlying physics of cosmic rays was revealed by analysis of the expanding 
data set. 

Research and development is currently underway on two detection techniques which 
could complement the array of water Cherenkov and air fluorescence detectors. The 
first is radio detection, a technique that was first pursued many years ago but is now 
benehtting from recent advances in electronics. The second depends on the possibility 
that microwave radiation, arising from molecular bremsstrahlung of electromagnetic 
shower particles on air, is isotropic and strong enough to be effectively recorded. If 
successful, one or both of these techniques could substantially extend the power of 
cosmic ray air shower detectors. 

16.1. Radio research program 

High-energy cosmic ray air showers generate radio emission via two processes: one 
is a geomagnetic, current-induced emission mechanism [152]; the other is a charge- 
excess mechanism [153], The observation of air showers with radio detection tech¬ 
niques can be done at all times (day and night). Moreover, radio signals are sensitive 
to the development of the electromagnetic component of particle showers in the atmo¬ 
sphere of the Earth and, in particular, to the depth of the shower maximum or mass of 
the incoming cosmic ray [154]. Radio detection of air showers started in the 1960s, 
and the achievements in those days have been presented in reviews by Allan [155] and 
Fegan [156]. More recent developments are based on initial studies performed by the 
LOPES [157] and the CODALEMA [158] collaborations and the LOFAR radio tele¬ 
scope [159], In the last 10 years the radio detection technique in the MHz region has 
been revived and the present radio detector arrays for cosmic ray research are equipped 
with low noise and high rate analog-to-digital converters. Simultaneously, the number 
of stations within these arrays has grown from less than ten to more than one thousand. 
The questions to be addressed in the VHF band (30 - 300 MHz) are: can we use radio 
signals to determine the primary energy, the arrival direction, and the mass of cosmic 
rays with accuracies which are equal to or better than those obtained by other tech¬ 
niques? If yes, can we build a large surface detector array based on the radio detection 
technique for an affordable price? 

The Pierre Auger Collaboration has started a research program to answer both ques¬ 
tions through a stepwise approach. Since 2009 the activities are coordinated within the 
Auger Engineering Radio Array (AERA), which is based on work within the Collabo¬ 
ration using various prototypes at the site of the Pierre Auger Observatory [160-162]. 
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As a first step, the emission mechanisms need to be understood. The contributions 
of the main emission mechanisms have been measured recently [163, 164], The frac¬ 
tion of the charge-excess emission relative to the geomagnetic emission varies between 
~ 5% for very inclined showers at 50 m from the shower axis to more than 20% for 
vertical showers at 250 m from the shower axis. As a second step the data obtained 
with radio detection stations deployed at the Observatory will be used to check their 
sensitivity with respect to the determination of the air shower parameters. We take 
advantage of the existing infrastructure of the Observatory: its surface detector, its flu¬ 
orescence detector and its low energy enhancements HEAT and AMIGA [131, 133]. 
To help answer the questions, hardware and software are being developed to study the 
required specifications and performance of solitary radio stations as a blueprint for a 
large array. Within the same research program, a rigorous effort has been and is be¬ 
ing made to understand the emission processes using our current knowledge of the 
development of air showers. Simultaneously, experiment and theory are being con¬ 
nected through software tools where end-to-end simulations and data analysis can be 
performed within the same software package [69]. 

The scientific goals of the AERA project are as follows: 1) calibration of the ra¬ 
dio emission from the air showers, including subdominant emission mechanisms; 2) 
demonstration at a significant scale of the physics capabilities of the radio technique, 
e.g., energy, angular, and mass resolutions; and 3) measurement of the cosmic ray com¬ 
position from 0.3 to 5 EeV, with the goal of elucidating the transition from galactic to 
extragalactic cosmic rays. 

Each radio detection station is comprised of a dual polarization antenna, sensing the 
electric field in the north/south and east/west directions, associated analog and digital 
readout electronics, an autonomous power system and a communication link to a cen¬ 
tral data acquisition system. The antennas are sensitive between 30 and 80 MHz, cho¬ 
sen as the relatively radio quiet region between the shortwave and FM bands. AERA 
deployment began in 2010 with 24 stations. These stations are equipped with logarith¬ 
mic periodic dipole antennas and are connected via a fiber optic link to a central data 
acquisition site. Stable physics data taking started in March 2011, and the first hybrid 
detection of cosmic ray events by radio, fluorescence, and surface particle detectors 
was recorded in April 2011. In May 2013 an additional 100 stations were installed. 
They are equipped with butterfly antennas. Detailed simulations and measurements 
demonstrated that butterfly antennas perform better for narrow pulse detection as com¬ 
pared to the logarithmic periodic dipole antennas [165]. The additional stations are 
connected through a wireless communication system to a central data acquisition sys¬ 
tem. AERA successfully deployed 25 additional radio stations in March 2015. AERA 
now includes 153 radio detection stations, spread over an area of 17km 2 . The detec¬ 
tor spacings range from 150 m to 750 m, which enables the full exploitation of radio 
detection of air showers as envisioned in the AERA technical design report. 

16.2. Microwave research program 

Recent results of a test beam experiment at SLAC [166] showed that it could 
be possible to use microwave radiation to detect extensive air showers. This radia¬ 
tion, expected to be isotropic and broad in frequency, is interpreted [166] as molecu¬ 
lar bremsstrahlung (MBR) produced by the scattering of low energy electrons in the 
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weakly ionized plasma produced by the shower with the neutral molecules of the at¬ 
mosphere. The Auger collaboration is pursuing an active R&D program to determine 
if a detector sensitive to MBR would be a suitable alternative for the study of ultrahigh 
energy cosmic rays. 

This R&D program [167, 168] consists of three different setups installed at the Ob¬ 
servatory. The AMBER and MIDAS experiments use radiotelescope style detectors 
intended for the observation of the shower longitudinal development in the same man¬ 
ner as an FD. In the EASIER setup on the other hand, SD tanks are instmmented with 
smaller radio receivers that take advantage of the enhancement of the signal when the 
shower is observed close to its axis. 

Installation of the microwave detectors was finalized in September 2012. A previ¬ 
ous result by the MIDAS detector [169], obtained in Chicago, places tight constraints 
on the amount of microwave signal emitted and its scaling with the energy of the 
shower [170]. The ongoing work to identify showers detected at the same time in 
the SD and in one of the microwave detectors already yielded the fist unambiguous 
detection of a cosmic ray shower in the EASIER setup in June 2011 [168]. 

17. Final remarks 

The Pierre Auger Observatory is the world’s largest cosmic ray observatory. The 
Observatory is highly productive, with very high efficiency data accumulation and so¬ 
phisticated atmospheric monitoring for accurate interpretation of showers. There have 
been recent advances enabling the extraction of longitudinal development information 
from surface detector information alone. New analysis methods using both the fluores¬ 
cence detectors and the surface array have improved the determination of primary com¬ 
position. At the same time the recently completed HEAT and AMIGA infilled enhance¬ 
ments now extend the reach of the detector to cover the critical galactic/extragalactic 
transition region. The Observatory is also serving as a test bed for advanced detector 
technologies for the next generation of cosmic ray experiments. 

Current source scenarios assume that particle acceleration takes place at sites dis¬ 
tributed similarly to the matter distribution in the universe, with energy loss processes 
leading to the observed flux suppression (GZK effect) and arrival direction anisotropy. 
However, Auger data on shower development fluctuations, as well as other composition 
sensitive observables, require consideration of a rather different interpretation: that the 
observed flux suppression is indicating the upper limit of the power of the accelerator. 
It may be that the upper end of the cosmic ray energy spectrum is dominated by par¬ 
ticles from a single source or source population, possibly within the GZK horizon, for 
which the upper limit of particle acceleration almost coincides with the energy of the 
GZK suppression. To answer this question, the Pierre Auger Collaboration is planning 
an upgrade to the Observatory to enable a determination of the primary composition on 
an event by event basis at the highest energies. The required electron-muon discrimi¬ 
nation in the surface detector stations will be achieved by installing a 4 m 2 scintillation 
detector on top of each water Cherenkov detector. Once approved, the upgrade is ex¬ 
pected to be completed by 2018 with Observatory operation through 2025. 
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